Robbie Clutton

Notes from "Blending MongoDB and RDBMS for eCommerce"

2011-09-19T14:48:00.000+01:00

github.com/spf13
spf13.com
@spf13
OpenSky eCommerce platform

products in mongo using custom fields for different product types (i.e. actors in movie, tracklist for album etc)

- ordered items need to be fixed at time of purchase, table inheritance bad for this

## 3 things for e-commerce

- optimistic concurrency (update if current, then try again if document is not current)
- assumes environment with low data contention
- works well for Amazon with long tail product catalogue
- works bad for ebay, groupe, anything with flash-sales (high data contention)

# Commerce is ACID in real-life

purchasing something from a store deals with this without concurrency as each product can only be held by one customer

MongoDB e-Commerce

- each item (not sku) has it's own document
- contains
-- reference to sku
-- state
-- other meta-data (timestamp, order ref etc)

cart in card action difficult, but in Mongo changing state on item makes it unavailable to other customers (e.g. if state is 'in-cart')

## Blending

Doctrine (OS - ORM/ODM) modelled on Hibernate

in SQL
-- product inventory, sellable inventory, orders

- inventory is transient in Mongo collections
- inventory kept in sync with listeners

for financial transactions we want security and comfort of RDBMS

## Playing Nice

products are stored in a document
orders are entities stored in relational DB
store reference not relationships across two databases

jwage.com/2010/08/25/blending-the-doctrine-orm-and-mongdb-odm/

Notes from "Schema design at scale"

2011-09-19T14:46:00.000+01:00

# Eliot Horowitz

## Embedding sub-documents vs separate collection

i.e. blog post and comments

- embedded
-- Something like a million sub-documents is going to be unwieldy
-- need to move whole document (expensive) if gets large
- Not embedded
-- Million 'comments' in separate documents means lots of reads
- Hybrid
-- one document for core meta-data
-- separate document for n comments with array separated by buckets (i.e. 100 in each bucket)
-- reduces potential seeks (if 100 in each, reduces from say 500 to 5)

## Indexes

- Right-balanced index access on the B-Tree
-- only have to keep small portion in RAM
-- time based, object id, auto-increment
- Keep data sequential in index (covered index)
-- create an index with just the fields you need so you can retrieve the data straight from the index
-- index is bigger
-- good for reads like this

## Shard Key

- determines how data is partitioned
- hard to change
- most important performance decision
- broken into chunks by range
- want to distribute evenly but be right balanced, like month() + md5(something)
- if sharding for logs, need to think
-- why do you want to scale (for write or reading)?
-- 'want to see the last 1000 messages for my app across the system'
-- take advantage of parallelising commands across shares (index by machine, read by app-name)
- no right answer

## Lessons

Range query vs regex (that uses ^ - essentially 'starts-with') is about same performance
If you have a genuinely unique id, use that instead of the ObjectId

Notes from "Scaling MongoDB for Real-Time Analytics"

2011-09-19T11:17:00.000+01:00

* Scaling MongoDB for Real-Time Analytics, a Shortcut around the mistakes I've made - Theo Hultberg Chief Architect, Burt

40gb, 50 million documents per day

use of mongo
- virtual memory (throwaway)
- short time storage (throwaway)
- full time storage

why
- sharding makes write scale
- secondary indexes

using jRuby

mongo 1.8.1
updates near 50% write lock incrementing single number
pushing near 80% write lock

mongo 2
updates near 15% write lock
push near 50% write lock

iterations
- 1
-- one document per session
-- update as new data comes along
-- 1000% write lock!
-- lesson: everything is about working around the global write lock
- 2
-- multiple documents using same id prefix
-- not as much lock, but still not great performance, couldn't remove data at same pace as inserting
-- lesson: everything is about working around the global write lock
-- being more thought about designing primary key (same prefix for a group)
- 3
-- wrote a new collection every hour
-- lots of complicated code, bugs
-- enables fragmented database files on disk
-- lesson: make sure you can remove old data
- 4
-- sharding
-- higher write performance
-- lots of problems (in 1.8) and ops time spent debugging
-- lesson: everything is about working around the global write lock, sharding is a way around this (4 shards means 4 global write lock)
-- lesson: sharding not a silver bullet (it's buggy, avoid it if you can)
-- lesson: it will fail, design for failure (infrastructure)
- 5
-- move things to separate clusters for different usage (high writes then drop; increment a document)
-- lesson: one database with one usage pattern per cluster
-- lesson: monitor everything (found replica that was 12 hours behind, useless!)
- 6
-- upgraded to monster servers (high memory quad extra large with AWS) (downgraded to extra large machine - 6 machines with mongod 12gb ram, 3 machines for mongo-config, 3 machines for arbiters across three availability zones writing 2000 documents per second, reading 4000 documents per secondc)
-- lesson: call the experts when you're out of ideas
- 7
-- partitioning again and pre-chunking (need to know your data and range of your keys)
-- partition by database, new db each day
-- decrease size of documents (less in RAM)
-- no more problems removing data
-- lesson: smaller objects means smaller documents
-- lesson: think about your primary key
-- lesson: everything is about working around the global write lock

Q&A
- would you recommend?
-- best for high read load, high write load is not a solved problem yet
- EC2
-- you have replicas, why also have EBS?
-- use ephemeral disk, it comes included and is predictable performance (with caveats like spreading across availability zones to avoid data centre loss)
-- use RAID 10 if using EBS
- monitoring
-- mongostat
-- plugins that are available
-- mms from 10gen
-- server density (here at Mongo con)
- map reduce
-- moved away to be more real-time (but was using hadoop)

My first scala app

2011-06-05T18:37:00.003+01:00

Well I say first, but that’s not quite the whole story. I’ve worked on existing Scala code bases but recently got the opportunity to start a new project and I got to see how the experiences on those projects and conversations with colleges influenced design decisions. I wanted to note down some of the tricks I’d picked up.

Tools

We start with the tools and first up is SBT, version 0.7.x. We depend on the IDEA plugin to generate IDEA project files. We also use the Scaliform plugin which formats code every compile within SBT and the Sources plugin which when run downloads the source Jars without having to stipulate the ‘withSources’ directive on the dependencies.

The IDEA and Sources plugin are run as processors and are not checked into the SBT project file, whereas the Scalariform plugin is. The distinction is that a developer might choose to use a different IDE or not download the sources, but we should always try and conform to the same coding practices and being lazy, having a plugin do it for you is one less thing to worry about.

Inversion of control / Dependency injection

One of the interesting developments early on was the choice of the dependency injection frameworks. Would it be Spring, Guice or something custom? Actually, we decided not to have any and just use plain old servlets through our routing framework, Scalatra. I think this has actually lead us to write better code, for instance we have taken advantage of the companion objects for classes instead of relaying on repository classes that would need to be injected.

Routing

We started with the routing framework, Scalatra which is similar to Sinatra and other simple routing frameworks. It provides a clean way of exposing your application to the web and combined with the following, enables all of the methods to be only a few very focused lines of code. We used Scalate and SSP for the template rendering which can feel a little verbose with the variable declarations at the top, but when you can pre-compile the templates you can understand why those are required.

Databasing

All new projects are using MongoDB and we’ve been using the Scala driver, Casbah, for a while but this gave us an opportunity to try out Salat the case class wrapper. This is probably as close as I’ve seen ORM like features in Scala while maintaining the MongoDB query sugar. As mentioned, using companion objects instead of repository classes allowed us to do tidy things like:


case class Artist(_id: ObjectId = new ObjectId, name: String) {
 def save() = Artist.saveArtist(this)
}

object Artist extends SalatDAO[Artist, ObjectId](...mongo config...) {
 def saveArtist(artist: Artist) = update(Map(“_id” -> artist._id, artist
}

And use them like:


val artist = Artist(“My new favourite band”)
artist.save()

This meant we were using a repository like model, but hiding it behind the case class and the companion object. It makes for very neat code further upstream.

Webifying

The app has to call out to other web APIs and for this we use dispatch which wraps the Apache HttpClient project with a bow making it really simple to get resources and convert them to string, XML or JSON. We were already familiar with lift-json so we would get the response as a string and use case classes to parse the JSON into. Once again, XML is almost a joy to work with in Scala compared to other languages, but lift-json still wins in simplicity so we use JSON formatted APIs where we can.

What’s next

I’ve got my eye on Lift and Akka. I’ve had a play with Akka, but I feel I’d like to understand it a little more before using it in a project. Other than that, I think I’m generally getting my head around Scala and functional programming all the more and I’m rather enjoying it. I still feel like I’m just scratching the surface, but it is my first app remember?

Building Curseleague.com

2011-04-10T13:24:00.002+01:00

I really wanted an excuse to play with NodeJS once again and after a conversation with blagdaross in the office I had my idea. I had also been looking for an excuse to stretch the legs of Joyents no.de hosting service for NodeJS. I had all of the pieces needed:

Twitter API

Cursebird API

no.de hosting

Once again I turned to the trusty toolset of ExpressJS for routing and EJS for template rendering. I wanted to build straight out of Twitter lists, so first thing was to hit Twitter to get the members of a list. As this is an open call, no oAuth handshaking has to take place, which is nice. However, getting the list of members returns a whole bunch of metadata about each member on the list. I didn't need any of this information, and I couldn't find all of the members without pagination. This is where I started to have some fun with writing asynchronous code for very much a synchronous web request.

Once I had the list of members of a list, I could do a lookup on Cursebird for the points they have given those members. I have to say the Cursebird API is just fantastic! Talk about the API being the website, just stick .json on the end of any page you visit and get the data you need. No API key nonsense for open data is also a great move. Well done to Cursebird on their efforts. After sorting the list according to points, all it takes is rendering.

I recognised that crunching all of this data with potentially a few hundred server side calls to Twitter and Cursebird would be slow, so I introduced a really dumb key-value pair cache and use that cache at every level, from caching a single request to Twitter or Cursebird, to caching a whole dataset for a webpage.

Onto hosting, and no.de has this really interesting deployment technique where you do a git push into a repository you're given on the virtual server you have. From there a post-push hook is fired which deploys your app and runs a named file (server.js). The no.de real time analytics are really impressive, although it'd be great to get historical information out of that too.

The one on-going grievance I'm dealing with is locally I deploy on port 3000 but in production I deploy on port 80, and currently I'm changing this on commits as a work. I had a way of doing this through a node module I'd built a while ago to get properties from a json file, although up until now I'd been using nDistro to manage my dependencies, but with no.de I'd have to do this a little differently. I could push the properties project into the NPM repositories but I'm also thinking of building a node module that can read nDistro files and add dependencies at run-time. It's food for thought.

Finally a big thank you to JT for working up the styling.

Cheers,

Robbie

Guardian SXSW Hack Review

2011-02-14T14:37:00.005Z

What I built and why

I really wanted to scratch my own itch at the hack weekend with SXSW in mind. There is going to be over a thousand bands playing at the music festival, and many of those will be trying to break through and make it. This means, there’s a fair chance I won’t have heard of many of the bands. Also, the sheer number of bands playing means it’ll be difficult for me to do any quick research about those bands to find out who I should go and see play.

I’ve used Last.FM for nearly four years and have something like approaching 20,000 scrobbled tracks in their dataset, so they have a good impression of the type of music and artists I like listening too and I wanted to tap into that data. Bearing in mind what I just said about not knowing any of the bands playing, I needed a different way to look at the data. Using Matt Andrew’s band listing API, I used the Last.FM API to find around 20 similar artists for each band playing at the festival, leaving me with a dataset of about 20,000 artists that are like those playing at the festival. My thinking here was there would be a good chance of bigger bands would be amongst this dataset and I might have more of a chance of finding a match.

Now, using the top artists from the Last.FM API, I could do an intersection of the artists I like and artists similar to those playing at SXSW. I could then do a look up to see what bands are similar to bands I like, helping me discover new music at the festival without investing too much time doing any research on all of the artists. Yes, it gave me a bit of a headache too.

I only really got this working by lunch time on the second day, and although I had intended to build a simple HTML layer on the data I’d built, I just didn’t have the energy. I went for a coffee, a chat and a nice sit down instead. I don’t think it helped tremendously with the presentation not having any visuals, but I think a few people saw the potential of the application and the data behind it.

What’s next

The hack day is done, but I want to push this project on a little more. The code certainly needs a lot of love and I wasn’t using a complete dataset over the weekend, so I need to import that to get more comprehensive results. I’d also like to make this more generic for any festival, be it SXSW or Glastonbury.

Lessons, observations and stories

The hack weekend was only my second hack event, and the first one I went too was only for the first day of a two day event, so I suppose this was my first full hack event. I’ve been to plenty of small meet-ups, larger Bar Camp events and mammoth conferences, but a hack day definitely has a different and more intimate vibe too it. The number of attendees seemed pretty optimal and it was certainly a good mix of developers, designers and journalists although this only really shone through in the final presentations.

Although having an idea of what I was intending to build at the event, and even doing some thinking about how the application might work (but no coding before the pistol on Saturday morning, as that’s cheating!) it still took me a while to get started. I probably would have coded my idea up in Node.JS as it’s quickly becoming my language of choice in the ‘get something done fast’ category, but as I was building something that I hoped would eventually be brought into the Guardian for festival coverage I thought I should build the application in the fast becoming language of choice at the Guardian, scala.

Scala, although fantastically awesome on one hand, still runs on the JVM and still requires Java like setup with web.xml files and the like. This isn’t a problem in itself, but it means some non-trivial time is spent just setting up a project the way you like it.

This leads nicely into lesson one:
If intending to use a language that requires some investment in setup, look for a way of reducing this, perhaps by having a library of “hello world” applications pre-configured for repeat use.

Keeping with Scala, as a relative newbie to the language I know I was doing certain things in an inefficient way and that was compounded by the timescales of an event like this. When working on a project and you suffer a setback, either you don’t know how to do something or something you thought would work just doesn’t, you generally have time to investigate, ask around and try a few things out. However, that is a real luxury when trying to build things in hours and not weeks.

Lesson Two:
If you’re going to use a hack day to experiment with a new technology, expect frustrations and delays. Even a half hour delay can feel catastrophic.

Just to finish the Scala points, and this may be my lack of knowledge of the language, but I wish the JSON and HTTP support were a little better. Compared to the XML support, which is an excellent xpath like implementation, the JSON support felt clunky. I actually had to change the data I was getting (I was reading from a file, so I could do this) to remove some parts which I couldn’t get the code to parse. As for the HTTP support, I had to bring in the Jetty HTTP client (which didn’t seem to recognise ‘utf-8’ as a character encoding), then bring in the Apache Commons HTTP client to request data from the Last.FM API. One post on Stack Overflow I was reading while I was looking for answers to a Scala problem suggesting having a personal library for wrapping functions you wish were supported better.

Lesson Three:
Knowing how to do common things really well, and fast is essential. In my case, using web based APIs using JSON and rendering JSON in turn.

One technology I did use which I’m now using on a day to day basis is MongoDB and this is where knowing something about a technology really came into it’s own. Getting stuff into and out of the database was as easy as it should be. I used the effective but perhaps slightly noisy Casbah Scala driver to talk to MongoDB. I was also using the Last.FM API to get information about bands and I realised that the Last.FM API was probably one of the first public web APIs I used thanks to a Paul Downey workshop when I was a fresh faced graduate. In a good way the API doesn’t look to have changed, which should be great as anything I built in that workshop might have a chance of working today. However, the web has moved on a bit from RPC and XML and I’d like to see Last.FM offering more of a RESTful JSON based API. That might just have influenced my decision to use Node.JS instead of Scala due to the support of JSON over XML.

As this was a music project, the sensible thing was using Music Brainz ids for the bands, and although this worked in many instances some of the bands playing SXSW don’t yet have Music Brainz ids and perhaps more surprisingly the Last.FM API doesn’t seem to provide Music Brainz ids for all of the bands in it’s API, even top 100 chart bands can be without one. The algorithm I built depended on this data, and although I could do a best effort and call Music Brainz directly, it would have been nice if this was covered off by the larger provider, Last.FM.

Update: I've already found out Last.FM does actually offer JSON in it's API, that would have been handy to know, and sorta proves the lesson of know what you're doing before starting.

Cache me if you can

2010-12-20T17:20:00.001Z

From the producers who brought you 'Tengo and Cache', we present 'Cache me if you can'.

The concept is all about HTML delivery into iOS devices alongside a mechanism for updating the application without going through the Apple app store. Loading HTML from a local file is not a new idea, as a few companies and projects have sprung up around this type of mobile app development have emerged, most notably Phonegap and more recently Apparatio.

The original iOS app I had built, Tengo and Cache, created a way that took this idea in a slightly different direction. The projects above deliver HTML within an application and would rely on Javascript to update any content within the application, using local storage to persist data. What Tengo and Cache offered was a way to download new HTML documents in their entirety, storing the files in writable areas of the file system on a device. This worked by providing a manifest file, based on the HTML5 manifest, on the domain which was the iOS app was trying to download files from.

During a hack day for the Guardian, I extended this work to add a further downstream cache. Although every effort is made to cache resources before loading, some files may be requested which are not included in the manifest. Overriding the NSURLCache class, the application can intercept any calls which attempt to go out to the web. This cache then retrieves the file, stores it, and then serves this instead of continuing down the pipeline. The next time this file is requested, the cache serves the file on disk instead of hitting the web at all.

Now that I had an iOS application that should pre-cache and intercept, I needed some content to install onto a device. I had wanted to use a Wordpress blog, or RSS feed, but felt that this content could prove difficult. Without knowing what content would be delivered, I couldn't build a manifest file that would capture the entire content. Also, I thought it might prove to be bad user experience if something that would work on a web page when online might not work offline. Even something as ubiquitous as search might look to have failed miserably.

I decided to use something I knew I had control over, the Guardian Content API. Using a small NodeJS application, I could retrieve a query and present that as HTML, alongside an appropriate manifest file. Then all that was required was a small property change in the iOS app and a native application was ready to be launched.

Cache me if you can

View more presentations from Robbie Clutton.

Samsung Galaxy Tab - Review

2010-11-23T18:05:00.002Z

The first thing you notice when you start using a Samsung Galaxy Tablet is that it clearly thinks it’s a phone. Most people I show the device to also think it’s a phone as they do their best Dom Joly impersonation. I had only intended to use the device over wifi but I’m constantly reminded that I haven’t put a SIM card in, and that the phone can only make emergency calls. I noticed at one point that a significant portion of battery life was being put to use on phone related activities. At that point, I put the device into flight mode and then enabled wifi to try and make the battery last longer. Trying not to use the device as a phone means many of the Samsung applications simply don’t work as they require a SIM card, although why they need the SIM card I couldn’t tell you.

One of the things I was looking forward to doing with the device when I got it was to use it around the office as I’m using a desktop and I wanted a way to take notes, look things up and do demos. My first snag there was that the electronic keyboard is just too small for any note taking at length and when I held the device in portrait mode I was typing by thumbs alone. I’ve found that the screen on the Galaxy is just about right to web pages, and due to the size, I can hold the device in one hand, not unlike the Kindle. Many websites are redirecting me to their mobile version, which although can look very nice on the size of the device, it sort of deflects the purpose of having a tablet as opposed to a phone.

Some of the plain oddities are exposed when trying to use Google Docs on the phone. Having sent you to the mobile version, you can edit documents and spreadsheets, but not presentations. When trying to change to desktop mode you get a bizarre error about the browser not supporting web word processing. There was me thinking that all that was needed was HTML rendering, Javascript processing and an Internet connection.

I have felt I’m getting benefit when you consider just about how ‘cloud-enabled’ the device is. I’m yet to plug the device in to my computer, apart from trying to deploy an application I’d built onto it. For music I’ve been using Spotify, Last.FM or iPlayer for Radio 6. Tools like Google Voice have let me find, download and play podcasts without needing desktop software like iTunes with varying success. The speakers on the device are just about good enough to carry around the house with you, or plug the headphone jack into stereos in different rooms. I’ve used DropBox to drop video files onto the device and they just play without having to configure anything or install any codecs.

I’m generally enjoying being able to think of this as more of a computer than a locked down device though, by seeing the running applications and being able to navigate the file system. However there seem to be simple things I just can’t find, like changing the auto-lock timeout or being able to wake the device from being locked other than pushing a button at the top of the device, when my hands tend to be at the other end of the device.

I’m happy that Apple has some competition in the tablet market, but I think the experience needs to improve a lot, through both the physical device and the software that powers it. I’d suggest a good start would be to get the device to stop thinking it’s a phone. The Amazon Kindle has a SIM card for 3G access, but it’s knows it isn’t a phone. It may be a small point, but I prefer my devices without the split personality.

nodejs, ndistro and git submodules

2010-09-11T14:21:00.008+01:00

On the (very early) tapas platform at theteam I got a little stuck when mixing nDistro and git submodules and wanted to explain what I'd done to get around those issues. On the tapas-models module, MongoDB is used for data storage and tapas-models uses Mongoose for MongoDB integration, but nDistro doesn't download Mongoose's downstream dependencies

This is because nDistro downloads the tarball of the project from Github and that tarball doesn't include dependencies. It might be nice for Github to do this, and I'll search the Github Support site to see if I can find something about that. Anyhoo, the tarball doesn't contain it's git bindings, so I can't go in there and update the git submodules.

The unfortunate thing is I've had to expose myself to some of the Mongoose inner workings to get the dependencies, but once I know them, it's a light touch for the next part. Using nDistro as normal, I include that dependency, including the revision number in my .ndistro file. As nDistro is executed in a bash environment, shell scripting can be used alongside the module declarations. So, I use a Linux move command to put the dependency where Mongoose expects it and everyone's happy.

It's a simple solution to what was a nagging issue and keeps me on my happy path toward NNNN

Recipe: Guardian Fried Chicken

2010-08-19T20:47:00.007+01:00

This evening I made an attempt at the Guardian Fried Chicken, and although Tim Hayward gives the recipe in the associated video, I wanted to share how I had gotten on.

Although the recipe is simple, it does take some time.

Soak the chicken in milk overnight and cover with cling film

Poach the chicken in the milk it was soaked in by bringing to the boil and then simmering for 20 minutes

Meanwhile combine the spices and blend, then add to plain flour

When the chicken is cooked, leave to cool in the milk

Coat the chicken in the spiced flour

Return to the milk and dip again in the spiced flour

Place into hot (but not scalding) oil and cook for 5-6 minutes until nicely brown

Pat down with kitchen paper

I also added some BBQ beans (just normal beans with brown sauce and lots of pepper) and some coleslaw for good measure.

I also had to make some compromises. I don't have a fryer, so I used a pan. Without a temperature guide, I used a trick my Mum used to do which is to drop a small piece of bread into the oil to get a feel for how hot the oil is.

So generally it all went well, although I had the oil too hot, so the coating became too crispy too early and when cut, the chicken looked slightly underdone. I put it in the oven for 10 minutes to finish it off, so while the meat was cooked it also dried out the coating. However, as I was cooking the pieces one at a time, I turned the heat down a lot so that the 5-6 minutes fried cooked the meat all the way through.

The one thing that was missing from the recipe was MSG for the spices. It might have enhanced the flavour more, it was already pretty good. It still didn't have that KFC feel to it though. I think I'll be trying it again though.

Check out the set for more photos.

Tengo and Cache

2010-08-05T10:27:00.001+01:00

This project was born from a desire to push on from some work of creating iPhone applications using only web technologies that I had done some time ago. That work can be seen http://github.com/robb1e/iWeb. This first project was a way of deploying a HTML/CSS/Javascript application onto an iOS device and have it run as a native application.

The issue with that was the only way to change HTML, CSS, Javascript, images or other data was to depend on a live internet connection or do an application update. I was looking for a way to make it easier to cache those resources with a specific eye on offline experience.

After some thought, the lightbulb switched on and it became an immediate choice to use the HTML5 Cache Manifest file as a basis of discovering what resources should cached. This project aims to do just that.

Between the application bootstrapping and a webpage being rendered, the cache manifest is read and each file is downloaded and stored in the iOS application sandbox directory. After all files have been saved, the UIWebView will render an index.html file in that sandbox.

Check it out: http://github.com/robb1e/Tengo-and-Cache

Json and Grails

2010-07-08T13:41:00.003+01:00

Recently I was trying to return JSON data from a Grails app and I was having some issues. Grails has some built in conversion utilities to enable code like


render myObject as JSON

This is all well and good, but I have a few fundamental issues with this. First off, attributes like fully-qualified class name and the database primary key are returned. I don't really want to expose internal information to client side applications. It also doesn't allow any renaming of the attributes. Say you have a String called firstName using camel case in your domain object, it returns that to the client. What if you want the API to return first_name?

The answer was a lot simpler then my searching could muster, and that is to use the template system, Sitemesh in this case, to render exactly what you want in JSON as you would if you were rendering HTML. OK, it's a bit of a headache, but you have complete control over the content and style. Combine that with a one liner in the controller to set the content type, and you're flying.


response.contentType = "text/javascript"

Spider and Validate

2010-06-25T13:27:00.005+01:00

One of the things we've been trying to achieve is automation is every sense. As my old mate Otu once said "automate until it hurts!" Well, hopefully this will make it a little less painful. We wanted to point to a single page, have the application spider the entire site finding pages which return anything apart from 200 OK, and validate all those who do return 200 OK.

By combining two PHP projects, PHPCrawl (http://sourceforge.net/projects/phpcrawl/) and the frontend-test-suite (http://github.com/NeilCrosby/frontend-test-suite) it's been possible to do just that. Throw a little Ant build script in there too and deploy into a continuous integration container like Hudson, and you start to get a feel for the state of websites in development, or even in production as part of a monitoring tool.

To make use of this, edit the build.xml file to change the SITE_URL to be the endpoint which you want to test and run 'ant validate'. This will do an Ant copy with filtering to create a file called test.php. Ant then runs this and then captures the output from this. It collects which pages may have returned a 500, a 404 and 200 OK pages. It then passes an array of 200 OK pages into the frontend-test-suite which uses PHPUnit to report on HTML validation.

There's still plenty of work to do, mainly introducing the ability to supply your own W3C validator endpoint and introducing CSS validation. It'll come though, so keep an eye out!

The result of this is over at Github - http://github.com/robb1e/Validator

As a side note, to get frontend-test-suite as a submodule on Github I followed instructions from FND, thanks!

Grails multiple web sites, same web application

2010-06-22T07:44:00.004+01:00

We had a need recently to have one web application serving many web sites, where the same data schema was in place with data shared across multiple web sites, but we wanted to show different styling and different data depending on the URI. In the end it turned out to be fairly simple. With our naming conventions, it's possible to hit one of several URIs for the same site, such as:

* project.testing.com
* project.staging.com
* project.com

So I had created a domain lookup service that when passed the request URI could work out what site the request for targeting.


def targetUri = request.getServerName()

I did a lookup for that target name against an object stored in the database relating to that targetUri and passed that back, making it available for queries for the correct data to show, and passing that down into the views to render the relevant CSS and Javascript files.

Grails, Ant, Hudson and automated deployments

2010-06-17T10:23:00.009+01:00

One of the things I like about Grails is the embedded server, it makes deployment and getting things up and running nice and simple. I've tried building a WAR and deploying into Jetty and Tomcat, but for some reason the system resources seem to spike when deployed in this fashion compared to the built in 'run-app' functions of the Grails command line tools.

With this in mind, I wanted to use the Grails command line to do automated deployment based off our Hudson continuous integration container via Ant build scripts.

First I created an init.d start/stop script for the server. At theTeam we can be working on multiple projects at the same time, so we need to set some conventions about port usage so we can manage our applications easier, so this is the first step towards just that.

This script is simple enough, on start it checks to see if that port is used, if so it bombs out, if not it nohup's the Grails command, telling it which port to use, and also just for ease of management, adding the name of the site. This means running commands like ps -ef and pipping to grep shows what's running. Stopping searches for a process running with that port and kills it.

That's the first step. Next is the Ant script to deploy the latest code after tests and static analysis. In this, the target folder has the latest code, we stop any server that's running, do a Grails clean and then run the server. The script then waits to see if it can open an socket to the port that the server is running on, and fails it it cannot. This usually means the server has failed to startup. The variables are gathered from a properties file and should be fairly obvious.

Now, you might have noticed a few oddities in the script like setting JAVA_HOME and BUILD_ID. First, when running in Hudson, any processes are run using the JRE and JDK that is deployed with Hudson, and I wanted to ensure it was the system Java I had installed. The BUILD_ID is a little trick that tells Hudson not to terminate the process after the build has finished, which it had a habit of doing in earlier versions.

When starting Hudson, a flag should also be set to tell it not to kill child processes. My start command is thus:

Adding this all together means it's possible to get automated deployments with Grails added into your builds.

AWS S3 Integration with Grails

2010-06-17T10:21:00.001+01:00

I’ve been playing with the S3 plugin for a Grails project and wanted to share how I set it up, as the documentation on the site doesn’t cover an example.

First, in my model I added a byte array field and made it transient. Transient basically means it doesn’t add the bytes to the database, but just keeps the data in memory. This means you can take advantage of all of the automatically generated controllers and view classes which are created from the model without having to really deal with the contents.

class ImageHolder{
byte[] image
String imageUri
static transients = ['image']
}

In the controller we use the request object to get the file and save it to disk, removing any spaces that may have crept in. Then we start using the plugin, and create a S3Asset from the file and put it into the asset service class. The assetService must also be declared at the top, and it’s injected via Spring from the plugin.

def s3AssetService
....
def imageholder = new ImageHolder(params)
....
def mhsr = request.getFile('image')
def fileName = mhsr.getOriginalFilename().replace(' ','_')
def ext = fileName.substring(fileName.lastIndexOf(".")+1);
def file = new File(fileName)
if (!mhsr?.empty && mhsr.size < 1024*200){
mhsr.transferTo(file)
def asset = new S3Asset(file)
asset.mimeType = ext
s3AssetService.put(asset)
imageholder.imageUri = asset.url()
imageholder.save()
}

What this does, is now uses the async service to put the file into S3, so your request returns very quickly. You can see that the image URI is now persisted into the domain object, but the content isn’t.

One final thing is to configure AWS in config.groovy. I set the flag to not add a prefix to the bucket name, as I’m using reverse DNS/package style bucket name to ensure uniqueness.

aws {
domain="s3.amazonaws.com"
accessKey="YOURACCESSKEY"
secretKey="YOURSECRETKEY"
bucketName="your.bucket.name"
prefixBucketWithKey=false
timeout=3000
}

Apple, standards and plugins

2010-06-16T07:11:00.009+01:00

I hate to sound dramatic but with Apple sounding off about web standards over plugins with their latest products I was wondering just how much Apple was dragging he industry forward, or if it was doing the opposite.

Some years ago, developing web applications meant having to know the details about the browsers your audience were using. We're still suffering from this today with discussion over different box models and questions about just when are Microsoft going to release a standards compliant browser. One example of this is anyone who uses Exchange for email. They might know the pain of using the web client given how rich it is in Internet Explorer and how average it is in any other browser.

The advent of cross browser JavaScript libraries helped reduce many of these pains and it's great to see a lot of work going into the new HTML5 specs that are taking lessons direct from these libraries. However, whilst singing the praises of a few websites that are using the new HTML5 audio and video elements, Apple seem to be creating their own events for touch that I can't see anywhere in the specs. It seems Mozilla are also playing the same game also and I'm worried that we're returning to a world where we have to tailor lots of code to target specific browsers.

You only have to go to Nike.com or log into Gmail to see iPad websites that have orientation and swipe support. It's all very nice but how much extra work is going to have to take place to get a website to work in different browsers and different devices? What happens when a new device comes out with different events, or even as with the new iPhone and it's different screen resolutions, how will that change the work already in the wild?

This raises further questions for me, especially after reports of JavaScript execution speed on the iPad , even though some frameworks claim this isn't so much of a problem, such as Spoutcore Touch
and others go further saying that no framework should be used at all to ensure performance

Although Apple have created fairly complete documentation about building web content for their devices, it does beg the question why there should be differences if we are supposed to live in a standards world? How long will it be before you hit a webpage and have it say "The website is best viewed with iPad", or any other device or browser for that matter?

Should we ignore the devices ad go for standards like the BBC iPlayer for big screens like PS3 and iPad , use libraries to augment sites like how apple.com/iphone allows the user to swipe the image carousel, tailor something specific for each device such as Nike.com or should we just be building native applications?

How should we progress? Should we let our analytics decide? Should we take the standards route or device specific route? This will certainly be one to keep an eye on.

iPhone development using standard web technologies

2009-09-05T12:39:00.008+01:00

Fresh after giving my talk from BarCampBright4, I wanted to get my notes down from my session on iPhone development using standard web technologies. This all started when I wanted to do an iPhone application for my Latest Scores Twitter app. So I bought a book and made my way through it slowly, trying to decode Objective C before coming across the part I was really after, the UIWebView class.

What I ended up arriving at was the ability to load and render a HTML file with it's own resources imported such as CSS, Javascript and images. The real magic in Objective C is this:


NSURL *url = [NSURL fileURLWithPath:[ [ NSBundle mainBundle ] pathForResource: @"index" ofType:@"html" ]];
NSURLRequest *request = [NSURLRequest requestWithURL:url];
[webView loadRequest:request];

What this is doing is creating a URL with a resource on the path called 'index' with the file extension 'html'. Build a request from this URL and pass the request into the instance of UIWebView.

From that point, as long as all of the resources that the index page requires are bundled into the project, you can load the page and have that actually be the application.

So that's all nice if you have a certain type of application which doesn't require any device interaction. For example, if you had a blog/news site and wanted to make it into an application, you could just get the content and display. You can also change a few settings in a file called info.plist to set the icon image and application name.

That template project is available on GitHub, http://github.com/robb1e/iWeb

However, if you want device interaction you'll need a richer framework, and that's where something like PhoneGap.com comes in. With the abstracted Javascript file that works for iPhone, Blackberry and Android, functions are available to get access to the contact list, audio playback and other device features.

To play an embedded audio resource for example, you can write the following Javascript, provided you've included the PhoneGap.js file.


new Media('beep.wav').play();

With both of these projects there is one caveat with Xcode in that it wants to compile Javascript files. You have to make sure you set those files as resources to copy, not to compile. This can be done in the target list on the left hand side.

To extend the idea of using standard web technologies whilst not building web sites, this next example uses Adobe's AIR run time and the HTML, CSS, Javascript and image from the first example. This is also available on GitHub: http://github.com/robb1e/AirWeb

By creating the application.xml file, you can tell Air to launch a file as the initial content and that will be rendered using it's WebKit engine.


<content>www/index.html</content>

To run the desktop app, install the Adobe AIR SDK then run the command:


adl application.xml

This launches the application in debug mode. To create an Air file, you need to compile using ADT.

This has shown that using the same technologies and languages you'd use to build a web page can be used for both mobile devices with and without device interaction and client applications that are installed on a computer.

There are other ways of achieving the same result using Apple's web namespace tags in the markup with HTML5 tricks like offline storage. There's also other libraries like jQTouch, so check out what else is around too.

The "write once, run anywhere" idea may just be coming into reality .

Metadata Cloud Gaming

2009-05-17T00:47:00.003+01:00

So, what the hell am I talking about? Let me give you a concrete example, and a personal wish list of mine for many a long year.

When I was growing up, one of the most capturing games was Championship Manager. This allowed the player to experience running a club, from transfers, tactics, training and more. However, the thing it always missed was being able to actually play the game. In terms of actually playing the game, when I was younger, the FIFA/EA Sports combo was hard to beat, although now I prefer the Konami Pro Evolution Soccer range.

I've always though that it'd be amazing to merge these two together, and I think now with Web2.0/Mashups/Cloud Computing and Web APIs this really should be possible. I could easily see a situation where as a gamer I could mash two of these together. For the tactical, and running of the game, I could use Championship Manager, and as a simulator for the actually games, I could use my game of choice. In this instance it really comes down to player stats. If I could load the game from Championship Manager, with the teams, players and 'stats' to boot, I could play out the game in my simulator and then upload the result along with accommodating stats into the cloud where my 'managerial' game could pick them up from, and vice versa.

Really, this is ripe for the picking.

The Great Twitter Camera Migration

2009-04-26T16:05:00.001+01:00

I've just started a fun little side project called the 'Great Twitter Camera Migration', check it out: http://cameramigration.blogspot.com

XFM Bot

2009-02-26T16:09:00.002Z

Just put together an XFM bot to scrobble the xfm.co.uk radio station recently played tracks and push that into Last.FM.

How compatible are you?

http://last.fm/user/xfmbot

Source code is here: http://github.com/robb1e/xfmscrobbler/tree/master

Vendor Driven Architecture

2009-01-30T16:55:00.006Z

So, you have your off the shelf application driving some important business process, maybe it doesn't quite do everything you need so you think about altering the application through plugins, changing source code or otherwise. Some of these changes may drive you further away from that off the shelf's product road map, making upgrading later difficult and expensive. Perhaps you don't intend to alter the code, but you may think that later on there may be a better off the shelf product that suites the needs of the business and that changing solutions is the best thing to do.

Depending on how these tools are used really drives the way these tools should be deployed and designed into a business process. If you only have one client talking to this application, you probably don't need to think too hard about how to solve this problem. However, if you have multiple clients wanting to talk to this application, you may want to give it some thought. If you have multiple clients, accessing the application in multiple different ways, then you definitely want to think about this problem.

I'm not an advocate of big design up front, but what I'm proposing isn't that exactly. What I'm suggesting is capturing the data model and contract of interactions early and use this as a way of driving an abstraction layer between the off the shelf product and the clients using that product. By doing this, you've isolated the application, meaning you can upgrade easier even if there are breaking changes, and you can introduce additional application logic all behind the safely of a contract and data model. After you've done this, anyone who uses your abstraction layer, need not know what you're using underneath. The way it should be really. This also defines a contract for working against, meaning that applications who talk to the abstraction layer can be tested easier against known behaviours.

We can start to push the boat out a little here and start thinking about roles and policy for access. There maybe different types of clients and users who wish to use the abstraction layer, and they may have different permissions and access to different resources, so how can you control access and protect your resources? You may be able to control that at the application layer, but what if you've added plug ins or additional application logic layers? And what if you change the application which manages permissions, and how do you bind them to the resources at the abstraction layer? It may be better to think in terms of the abstraction layer and data model rather than what's behind the lines of the abstraction. This way, binding to resources will be easier to control. While we're dealing with authorisation, authenticaion doesn't stray far, you'll want a way to authenticate the clients and users attempting to use the abstraction layer, and you may want that authentication to come from multiple sources, say a database user store, or an internal directory service. This will make it easier for those users to gain access to the systems, and easier to administer access through roles and policies of clients and users.

What we now have is a platform. An off the shelf application with or without additional application logic, wrapped with an abstracted and understood data model which must be accessed via an authenticaion and authorisation mechanism. Put a bow on it, you're done.

Depending on your network security preferences, you could now expand this in two ways. For every off the shelf product you employ for different business processes, you could put an abstraction layer on it with an understood data model and give access to that abstraction via the same authentication and authorisation layer you've created for the first platform, or you could replicate the authenticaion and authorisation layer and have lots of self contained, secured platforms. It doesn't particularly matter which is employed, but what should fall out of this is a common data model across several platforms, with a uniform authentication and authorisation layer.

It does raise the question of why you'd want to be doing this in the first place, you are after all creating more work than strictly necessary. There is a temptation to say "if we're building this abstraction layer, we may as well replace the off the shelf component itself". Well, let's not get ahead of ourselves. We have to remember where the business value comes from, and that's generally in products and services, not business support and operational support systems themselves. However, by abstracting the applications used and exposing the data, the cost of integration, migration and maintenance is lowered. This model can be applied to internal exposures and for exposing data on the web.

I just want to close with what I believe is the most important aspect here, and this is the business case. There is a business case for abstraction, but it does come at a cost. What it brings is contractual interactions and can lead to common, agreed upon resource based exposure.

2008-12-27T17:41:00.003Z

The fall of Zavvi along with several other soon to be notable high street absentees and the troubles the banking system has been going through has got me thinking.

First I want to tackle failure of business processes. Zavvi's main reason for going out of business, according to the press was a domino affect from Woolworth's going into administration. Zavvi relied heavily on Woolworth's wholesale as a source of DVDs and CDs. From the outside in, it seems Zavvi operated with wholesale distributors of goods rather than dealing directly with manufactures. There's nothing wrong with that of course, however, if you deal with a middle man who goes out of business than you're left scrambling looking for a replacement at the same costs. No doubt, if you have one middle man, you've probably aligned prices closely (albeit probably a given percentage above) the distributor.

Why wasn't this adequately highlighted as a risk? Why only have one supplier? If you must have one supplier, at least have a good idea of alternatives at the drop of a hat if that risk is realised.

This comes nicely into my second point, it almost seems that Zavvi have suffered a misapplication of the Toyota Production System in that they pulled stock into their store as to minimise inventory, and therefore waste. Keeping the time between ordering from the wholesaler to getting into the customers hands is a worthy goal to aim for. After all, unsold stock is not money in the bank. But again, this raises the risk of only having one core supplier.

Moving onto Woolworth's itself, I find myself asking what Woolworth's actually sells and the only answer I can fathom is that it's a general store. They never specialised in anything, meaning that they never had the buying power to lower their prices and pass the savings onto their customers leaving them with a breadth of stock. In my opinion that stock wasn't a breadth of high end stock, it was a breadth of medium to low end goods, odds and ends and overly priced electrical/entertainment goods. If I needed a new set of tea towels, brilliant, but for the floorspace they wouldn't be able to justify just selling bits and bobs like that.

This leaves me asking, who are your customers, and what differentiates you from your competitors?

Finally, how can you expect to survive charging the prices both Zavvi and Woolworth's seemed happy to charge? In both stores I could find a product, say a DVD, from a fairly well known internet only business, play.com, cheaper that wasn't even on sale than on the highsteet in post Christmas/liquidation sales. Play.com does advertise nationally on TV, radio and in print, and although I won't argue the brand awareness between Woolworths and Play.com, it's not like people couldn't find Play.com.

I guess after all is said and done, as a business you need to know your customers, know your competitors, know your business processes and understand your risks and how to mitigate those risks. On top of that, I'd like to add you should always be asking these questions, always looking for improvements and changes.

Just look at Tesco, boy have they changed! They seem to know what their customers want, and are constantly changing, adding new product lines in store and online.

Doing the Right thing vs Doing it right

2008-12-20T13:49:00.003Z

There was one lightening talk at XPDay that has stuck in my head, and I want to try to reproduce the discussion here. I'm sure I'll get some details wrong, but you'll just have to bare with me. The talk was on doing the right thing verses doing it right.

If you look at the image below, you'll see an XY graph, along the Y axis is doing the right thing, i.e. building the right solution. Along the X axis is doing it right, i.e. building the solution well. There was some studies into these metrics, but I don't know what it is, so I'm just going to use pseudo numbers.

In the lower left quadrant, you'll see doing the wrong thing, and doing it badly. The cost of the project is greater than the average IT project, and the revenue is lower than expected. In the upper left quadrant your doing the right thing, but you're doing it wrong. Your costs are still larger than average but you are making more revenue as you're building the required product. In the bottom right you're doing the wrong thing, but you're doing it well so your costs are lower than average, but your revenue isn't as good as expected. In the top right, you're doing the right thing and you're doing it well. Your costs are lower than average, and your revenue is higher than expected.

Now, of course, this is abstract and to be taken with a large pinch of salt, but there are some interesting deductions to be made. Firstly, we were told that according to the study, it's near impossible to go from building the right thing badly to building the right thing well. Looking at the rest, it seems intuitive that if you were doing the wrong thing badly, you'd want to do the right thing badly before doing the right thing well, but in actual fact, it's less costly and more efficient to do the wrong thing well and progress to building the right thing well.

So, where's your project on the quadrant, where do you want to be and how to you want to get there?

Software Development: By Way

2008-12-20T12:34:00.003Z

I tend to think the way libraries and frameworks are made could easily be applied to teams and projects. When building an application, generally you focus on solving the business problem with the best tools available. When you move onto the next problem, you might have that eureka moment where you find that there are sub parts of the new problem which overlaps with the previous problem. The fact that you've already solved this problem, means you'd be silly not to reuse that. Thus, you have identified a common functionality, and if you extract that into a new library, you have reusable code across more than one project. Brilliant.

The other way that you'd come across common functionality is through conversation. In true water cooler mannerisms, this will often be coincidental, but again, if you can identify commonality and extract that, you are deriving business value. Not only are you reusing something, meaning time can be saved and all teams benefit from faster delivery, but with more eyes on the same code, the chances of that becoming more robust grow.

So far we mentioned code, but this could just as easily be process and tools, like build scripts, deployment practices, server provisioning and lots more too. The key is to make discovery, and sharing easy. I think the key here is relationships and communication.

I think this model can be applied to projects too, where new projects are spawned to develop/manage commonality, this team doesn't necessarily have to consist of full time staff, it could be part time or ad-hoc, but there should always be a project owner/lead even if the project is sleeping. Anyone should feel they can contribute to the project, for the benefit of all the teams, even if there are full timers on the project.

I think, over time, this will form a network of loosely joined teams surrounding a handful of core projects, almost like atoms around a nucleus. I see these core projects being things like:

* build scripts, including testing tools and reporting
* logging and monitoring scripts
* deployment processes
* network and hardware management and provisioning
* data access layers (especially if all projects share common relations like customers)
* service access layers, for accessing hosted services
* user interface asset management
* hosted tools support, e.g. source control, software repositories, wikis, mailing lists, story tracking, continuous integration boxes etc

There's probably a bunch more too. But as I mentioned earlier, relationships are probably the most important aspect here. Establishing a community between the project leads and another for the wider developer group across all the projects will help, but you have to support the community, let it flourish. The more the community collaborates and communicates the more they'll drive towards commonality and reuse. Of course, people will have different ideas and prefer different tools, and that should be supported as long as the principle of spinning out or contributing to projects carries on.