Hugo+

Our economy is based on The Wealth of Nations, and it’s out of date…

2011-08-24T23:15:00.001+01:00

This is a post I’ve been meaning to write for some time, and have never got around to as it’s just too big a thing to get straight in my head. I’m writing something now (and it’s little more than the title) just because I think I should lay down a marker. Apologies in advance to anyone who was looking for something more insightful.

The summary is this: The Wealth of Nations by Adam Smith is the de facto bible of western economics, and its core tenet is the Division of Labour, which he studied through the prism of industrialisation and specifically the manufacture of pins.

Problem is, we’re moving beyond a classic industrial society – possibly to something identified as the Information Age. And in this age, classical economics no longer apply. This is most clearly seen in my own industry – software development, which works in precisely the opposite fashion: productivity increases as work is consolidated into a smaller group of more talented (and expensive) developers. The Division of Labour is not only unproductive, but positively destructive, as anyone who has scrambled through an offshore project can attest.

This paradox undermines many well-meaning corporate initiatives, and until we have a clear, and open, debate on this we are, frankly, a bit f*ked.

More to follow.

^[1] Notwithstanding the “Invisible Hand”, which is the other core message.

Yes, Facebook has lots of users. So what?

2011-07-07T20:43:00.001+01:00

Another day, another article about Facebook's global takeover. Yes, I know that FB now has 750m members, which is clearly a lot of people. However, the obsession with registered users feels very like the rush for pixels in the digital camera business a few back, or the stampede for ever larger app stores amongst the mobile vendors.

I don't care if FB has 1bn members (anecdotal evidence suggests that registrations peaks at apprx. 50% internet users within a country), a camera has 50m pixels or my mobile app store has 1m applications - if they're not my friends, my favourite apps, or the pictures still come out blurry it's a problem.

Google+ is the victim in the latest article - with the Facebook-Skype deal apparently sounding the death-knell for the new Hangouts video-chat service. I don't think it will kill it, and what is more, I think the niche audience that G+ is rapidly acquiring (albeit fanned by the limited access to invites) is more valuable than the mass of humanity represented by FB*.

In a similar vein, if I was a camera vendor I would pay a lot more to advertise on Flickr than Facebook (yes, I know Flickr doesn't have ads, it's a theoretical point), because people on Flickr care about photos, and people on Facebook don't. (I'm also sure there are hundreds of amateur photo groups on Facebook - but if they really cared they'd be on Flickr, as it's a destination defined by the quality of images available.)

I am sure that it makes sense for all commercial entities to have a presence on Facebook, and to invest in a "social media strategy", but they are supporting acts; the web is the superset of all properties, and will outlive Facebook.

* NB This obviously doesn't apply to the drinks brand WKD - whose natural habitat is clearly Facebook on a Friday night.

What agile means to me

2011-06-05T20:33:00.001+01:00

[Update: in response to some of the comments here and on HN, I thought it might be worth updating with a note on my personal experience with Agile, which appears at the bottom of the post.]

Agile (with a big 'A') has become so mainstream now that it has started to become the problem. An alarming number of people who espouse the virtues of Agile, and who quote the Agile Manifesto believe that Agile is a project management methodology, and that Agile really means SCRUM, XP, Kanban, and that it is embodied in the daily stand-up, whiteboards or writing requirements on post-it notes. I was once told by an Agile Trainer (LOL) that the correct way to phrase the requirement "we need advertising placeholders on the site, and some way to manage which ads appear where" was "As a User, I wish to be marketed to." Needless to say his company lost a $m project on the back of such BS.

As a result of all this, Agile (big 'A' again) has become a bit of an albatross - it doesn't really work, it doesn't deliver the benefits it promised, and it inevitably involves a lot of arguing amongst the team about who should be doing what, and when.

All of which distracts from the fact that agility (small 'a' this time) is a wonderful thing, that can be achieved, that does provide enormous benefits, and that should be encouraged in everything we (as an industry) do. Agile (small 'a') refers to team dynamics, delivery processes, the software itself (how easy is it to change), and operations. Everyone should aim to be agile.

So, a few lessons I've learned along the (hard) way:

The best way to achieve your goal is to start with the right people
If you've found the right people, let them get on with the job you're paying them for
The people with the best ability to plan the delivery are the people doing the work
A small team with a manageable backlog needs very little Project Management
Everyone on the team “needs to know”.

A few things that help this happen on a practical level:

Try and avoid building towards a pre-determined marketing / launch date
If you do have a fixed time, then scope is variable
Never try messing around with team sizes to massage delivery
Automate everything in sight - builds, deployments, testing, progress reporting, tea-making
Measure as much as you can - no feedback == no direction

And finally some other things that will prevent you from getting there:

If you have to have everything on your requirements list, you can't be agile
If you need to plan beyond the next sprint with any degree of accuracy, you're not agile
If you think you can "fix" an iteration by adding more people, you're not agile
If you are prepared to change the end date of an iteration to "fit something in", you're not agile
If you have to give a fixed date for delivery - it's very, very difficult to be agile.

Postscript: my personal experience with Agile.

I worked for several years at a company that was at the forefront of the Agile movement in the UK, and I believe that at one point they had more registered SCRUM Master practitioners than anyone else in the country. (I think Sensai Schwaber visited in person at one point, although I’d left by then.) They were generally acknowledged to be the leaders in the field.

The incident in the post occurred when I returned to the company as a client, parachuted into a large project that they were running with the full SCRUM toolkit (“Agile Coach” included), that was running into trouble, principally the fact that it was not agile. The project was rescued by hiring a team of people with the right combination of aptitude & attitude.

As I say in the post – agility is a great goal, and everyone should aim for it – but don’t make the mistake of thinking that Agile is the means to achieve it.

Driving to the airport, agile-style.

2011-06-05T19:59:00.001+01:00

If you were travelling from A-B, and planning the journey yourself, you might look it up on Google Maps. GM would give you a route, and an estimated time for the journey - let's say it's one hour for our mythical trip. Now let's assume that the destination for our journey is the airport, as we have a plane to catch - i.e. we absolutely have to be there on time.

In this scenario, would you leave the house one hour before the flight departs (let's ignore the reality of parking / check-in / security etc. - we're on the company jet, just driving up to the plane)? Of course not - if you had to be there on time, and you were taking someone else's directions, even someone with the 'authority' of Google, you'd probably add at least 50%, and probably 100% to the estimate. Even then, you're still subject to the vagaries of the traffic - something that affects everyone, regardless of whether they've driven the route before.

If you were to approach the journey from an agile point of view, you wouldn't book the airline ticket in advance - you'd catch the plane when you got there. And you wouldn't plan the route either - you'd hire a driver, who knew the route, and simply tell him the destination. He'd make up the route as he went along, avoiding traffic problems if possible by using his previous experience. It’s pretty clear that if you just set out from A, travelling to B as fast as you could, it would take less time than if mentally started at B, then worked backwards to determine when you ought to leave A.

Agile behaviour in a software development team requires a similar faith in the ‘driver’ – in this case the people in the team. The quickest way to reach your destination is to have faith in the team to deliver, and to make sure that they understand the destination. If you then insist on them following a specific route, then you have just removed their capacity to avoid delays and dead-ends – removing the reason you hired them in the first place.

If you want to be agile, you have to trust in your team.

(And yes, this does mean that outsourcing a project cannot be agile. It also means that agile is NOT a project management methodology. It’s a way of working.)

Application Lifecycle Management

2011-05-16T00:49:00.000+01:00

[Update: a couple of people have called me out on this post, and suggested that it would only work from the bottom up if you hire good developers. This is true. This entire blog is based on the philosophy that the cheapest and most efficient way to achieve any given goal in software is to hire a smaller number of better developers. If you have a different philosophy, then you may find this site more helpful http://hrb.fm/keHPzD.]

ALM is described on Wikipedia as:

“…a continuous process of managing the life of an application through governance, development and maintenance. ALM is the marriage of business management to software engineering made possible by tools that facilitate and integrate requirements management, architecture, coding, testing, tracking, and release management.”

The phrase itself is enough to make the blood run cold, but I’ve been struggling recently to justify why I think it’s a Bad Thing, and I think I’ve worked it out: it’s all about the direction in which processes are created and applied.

ALM comes from the Old World (aka I.T.), where working practices are created by Architects and applied downwards to development teams. It’s a process designed to commoditise development, and to manage human beings out of the equation. If the process is sufficiently detailed and watertight, then developers can become replaceable – which in turn allows you to find cheapers versions.

This is lowest common denominator thinking – by creating a process that allows poor developers to work effectively, that same process will prevent good developers from doing what they’re good at. It’s mediocrity by design. Which in I.T. terms is not necessarily a bad thing – ERP, Accounting, Warehouse Management Systems – these are things that are: a.) operated in a controlled environment, b.) used by specially-trained staff, c.) defined by stability, not innovation.

None of this applies to web application development – which is defined in contrast by: a.) public access, b.) self-service, c.) continuous innovation, and pace of change. In this New World (otherwise known as The Web), companies are defined by the quality of their development staff – they are the asset, and they need be allowed to flourish, and not capped by process.

The kicker to this is that process is still a good thing in this New World – but the new processes are designed not to prevent developers from getting things wrong, but to make it easier for them to get things right. ALM should exist in practice, but it should (in fact must) come from the bottom up – the processes are organic and designed by the developers, for the developers.

There’s no requirement for management to even be aware of these processes – they should manage by outcome, not intervention.

NB If this rings any bells with you I would recommend Orson Scott Card’s “software developers as bees” article - http://hrb.fm/lIOR0w

Web app or native - looks like HTML is winning the war.

2011-05-09T14:42:00.001+01:00

When the iPhone launched, one of the major gripes was the lack of support for native apps. Steve Jobs launched the web-apps-look-like-native-apps program, was roundly criticised for the lack of a native SDK, and sure enough it turned out it was already in the pipeline. The App-Age was born.

(Not long after, Palm released webOS, a web-app platform from inception, and it was roundly praised for its innovative approach.)

Circle back round a few years, and the plethora of native platforms (iOS, Android, RIM, webOS, WP7) and form factors (phone, tablet) combined with the emergence of HTML5 as a "standard" (albeit one implemented (in)differently by each platform developer) means that web-apps are now back in favour.

PhoneGap in particular seems to have captured the mindshare of the mobile developer community; it will be interesting to see how this battle plays out over the coming year. Gaming aside, once local storage becomes a common feature, web apps look unstoppable.

Ode to Dropbox

2011-04-14T14:28:00.001+01:00

I’ve been using Dropbox for about 18 months, and before that I was a Mesh fan, and in the dim and distant past I spent many, many hours fighting with Plaxo and ActiveSync (see here for a post from 2004 on my frustrations at the time) - so I like to think I know a thing or two about file syncing issues.

I have a separate work Dropbox account, which for the past couple of months has been so far over its (free) limit that I've had to stop using it (there's only so much you can legitimately clear out!) In the meantime, two things have been happening - I've been adding to the local directory, in anticipation of a future upgrade, and other people have been adding content to the remote folders. I had enough faith in Dropbox already to be able to cope with this – I’d get another 3GB of new content coming down, and Dropbox would get a few MB of changes from my local machine. Easy.

Unfortunately, having upgraded, I discovered that in some overzealous spring-cleaning I had removed critical user profile files, and so the Windows user account associated with the work Dropbox account was corrupt, and I couldn't login and start Dropbox. This meant that all of my changes were now stuck in a local folder that couldn’t be synced, hidden amongst 2GB of data that had already been synced to the cloud.

My solution was to create a brand new Windows account and to copy over the entire 2GB archive to this new profile before hitting the sync button - but this meant that instead of adding just my recent changes, I would be syncing the entire 2GB local directory to the new user profile, 99% of which was already on Dropbox, uploaded from a different Dropbox sync. I was convinced this would result in conflict armageddon (and about 48 hours of file transfers – 2GB up, 3GB down, merge, sync, index.)

I left it running over lunch, came back and discovered... nothing. It had just worked.

Dropbox is an amazing piece of software, and if they don't sell up to Microsoft (Windows 8 - Live Mesh replacement), Apple (MobileMe replacement) or Google (Chrome OS file system) for several billion dollars something ain't right.

Google lives in Perpetual Beta, whilst Microsoft shoots for RTM.

2011-03-31T23:53:00.001+01:00

Something came up in conversation today that struck a chord with the (my) favourite topic of the moment – why is MSFT not responding to everything going on around it?

I don’t really know anything at all about MSFT’s business – I’m a .net guy, I’ve been to Redmond, I’ve seen the inside of the machine, but I don’t know all the ins-and-outs. But I do know what they do – and more importantly, how they do it, and that gives me enough to form an opinion.

They produce software. And the software they produce, even to this day, is distributed on physical disks. That’s right – they burn their software onto shiny plastic disks, which are then distributed around the world (“filling the channel”). And this is their problem.

Culturally, MSFT is still working in a physical disk world. If you’re burning your software onto 100m disks, you can’t afford to get it wrong, and you can’t easily get it back again once it’s left the building. So they create enormous product development cycles, building towards huge, monolithic, releases. They still call their final releases RTM (Release to Manufacturing). Their development processes are built around this concept, their product development process are built around this concept. Worse still, they have partnerships with disk-pressing plants and disk-selling retail channels. And all of this baggage they haul around with themselves to this day.

Back in the 90s, the talk was of how MSFT missed the boat, misunderstood the internet, and then woke up. They suddenly Got It. The built a better browser, killed Netscape, and now they owned it. Except that they didn’t. They thought that the internet was the browser – and since they owned the desktop, they could fold in a browser for free and their job was done. Wrong.

They still don’t get the internet; only 18 months ago I was being told, in Redmond, that whilst the internet was nice, desktop applications were nicer, so could we please put our internet-only Silverlight idea on hold, and just get with the Windows 7 program.

Google has none of this this heritage – it was born from the internet. As was Facebook. They can innovate and release at internet speed. A Google search results page is a continuously evolving multi-variate test. When they say that they live life in beta – they’re not lying – they really are trying new ideas every single day, live. Google is never “RTM”.

And Microsoft cannot compete – because of its legacy. They have a vault of cash, and some fantastic talent, but until they start to shed some of their physical dead-weight, until they acknowledge what the internet has done to their business (a business they invented), they are heading into retirement. The world they built is innovating away from them, and they’re just too slow to react.

webOS Developer Connect (London) – notes from a (very) small island

2011-03-30T23:25:00.001+01:00

I’ve just come back from the HP webOS “Connect” meeting in London, where a bar full of eager mobile web developers were being courted (with free* beer no less) by the worldwide VP of Developer Relations, Richard Kerris.

I didn’t stay the full distance, leaving after about an hour and a half, and that in itself is not a good sign.

Kerris himself gave a great, and sincere, introduction to the evening. He seems like a nice guy – competent, articulate, honest. He promised the best mobile developer program in the world, access to 100m devices in the coming years (HP have committed to getting webOS onto notebooks – or possibly netbooks, one of them). The devices look good, the OS seems to be right in the 2011 sweetspot, and HP has a mountain of money – so anticipation levels were high.

And then the presentations began.

I stayed for about half-a-dozen ten minute presentations, and frankly they were lamentable. Someone from the Guardian (where isn’t there a Guardian presenter these days) gave a great little talk on how important ~~iOS~~ mobile devices are to their future strategy. Unfortunately they don’t have any webOS applications.

Then came a German games developer – seriously, flown in from Germany. As if there’s no one comparable in the UK. Except apparently there isn’t. He too gave a nice talk – predominantly on how the first SDK didn’t really work, and how he ported his first webOS app to iOS in a couple of days. And he’s a paid (in expenses at least) advocate.

Then we had a succession of very nice people, all of whom had various degrees of hello world apps to show, but nothing commercially viable. At which point I had to leave.

I wasn’t the biggest fan of the Pirate at QCon, but at least he had a company, employing real people, making money from iOS.

I like Mr. Kerris, and I like webOS. Hell, I even quite liked an HP laptop I saw advertised the other day, but this was a pretty poor show from HP. They couldn’t even pull together a set of speakers willing to endorse the platform professionally. We wanted Rovio to turn up, or possibly Zynga. Or someone, anyone, who was doing something that might make money out of this. If HP are putting on a roadshow for developers (as is their stated intent) they need a lot more firepower, and if that means taking some US developers on the road with them, then that’s what they need to do.

* That’s free as in beer.

Color may have something to it after all…

2011-03-27T11:13:00.001+01:00

I was at an event on Thursday evening at @TechHub and the host for the evening invited everyone there to install Color and at least give it a go before wading in to the “$41m for what?” debate. His teaser was quite an interesting one – by firing up Color on his phone he was able to see photos coming in from the party upstairs, and by commenting on the photos (i.e. socialising) he got himself invited to the party.

Edge-case or mainstream – not sure, but it did at least counter the “wtf would I want to share photos with people I’m already in room with” argument.

(NB I still think $41m is absurd, and we are definitely in a bubble – Color may not be pets.com, but we’re only one high-profile flop from dotcom 2.0)

The Government is “joining up”– be afraid!

2011-03-26T14:54:00.001Z

Today I applied online for a new driving licence – and as part of the process I was warned that the photo they had on record for me was only valid for a further six months. However, when I put in my passport number it found the photo I had scanned in when I got my new passport a few weeks previously. I was asked if I would like to use that instead – as it was valid for the next ten years.

Aside from the great user experience (from a Government website no less), this points to two things:

There is now inter-department “joined-up” Government
I now have a de facto identity card – whether it’s physical or not

The ramifications of this are enormous, and - in conjunction with recent treasury annoucement re. the combining of tax and national insurance – we are closing in on a time when the Government might have a “single customer record” – which is not necessarily a good thing.

Using a foreign language to challenge UX

2011-03-23T12:15:00.001Z

I have spent the morning trying to test a new website that is currently available to me in Ukranian and Swedish – neither of which I speak. I have a basic understanding of the site, having seen it in its original english some time ago, and I know the functional requirements. Not being able to speak Swedish meant that when filling in a form I had to make educated guesses based on the pure UX itself – was the warning text in red, was it in a relevant position etc.

This turned out to be a fantastic way of testing the site. I repeatedly failed to submit a form because I hadn’t filled in a mandatory field, and the shape / style of the visual cues weren’t obvious enough to me.

Another example – something that was obvious to me, but not to the native speakers, was an inconsistency in button styles – which caused me to cancel my form submission because I thought the cancel button was the confirmation button.

I think I may have invented a new technique.

(In a similar fashion I do know someone who used to proof-read professionally, and she would read a document backwards – so that she could spell-check each word individually and out of context!)

How we use HipChat

2011-03-18T09:36:00.001Z

Inspired by a blog post (many years ago) by the 37Signals team on their use of Campfire, their own chat application, I have been trying, unsuccessfully, to wean the development team off Windows Live Messenger, Skype, et al. for team chat.

I had been looking for a long, long time for something that would allow us to maintain long-lived (and searchable) chat archives, but that would also allow us to plug in some of our automated systems. Campfire itself didn’t work for us as it’s web-based, and no one was willing to give up the taskbar icon / notification process.

Cue HipChat – which we’ve been using for a few weeks now, and which I am now unilaterally declaring a hit. I thought it might be interesting to show how we use the chat, and how our build / vcs systems interact with the chat application, so here you are – Profero developer chat, now in colour (click to view full-size):

QCon London 2011 Review

2011-03-14T16:56:00.000Z

I’ve done a quick round-up day-by-day, but here are my overall thoughts on QCon 2011.

The conference as a whole seemed slightly subdued after last year – it was sold out, so I guess it had the same number of attendees, but many of the sessions were half-full, and there was often an uncomfortable silence at the end of presentations when they were opened up for questions.

There was no stand-out presentation of the calibre of last year’s Facebook keynote, nor anything as technically impressive as the LMAX talk, despite presentations from Facebook, Google, Twitter, Netflix, VISA, Best Buy and a host of other big names. This year seemed to be more about getting things done – the nuts-and-bolts rather than the vision.

On the plus side, there was an entire track devoted to NoSQL this year, and many of the other presentations included large slices of NoSQL goodness – be in no doubt, NoSQL is no longer a niche – this is mainstream technology, in production across the web today. A big thank you to Alex Popescu not only for hosting the track, but also for his tireless efforts to bring NoSQL to the attention of the community with his blog http://nosql.mypopescu.com/

I’ve touched on it my daily digest, but I got a real feeling that some of the more extreme team dynamics (see Dan North and Fred George’s talks in particular) are gaining traction – there seems to be a split forming between the radical side which is eschewing almost all project management practices, and those who are artlessly turning Agile into everything Waterfall had been (process over people).

Maybe continuous deployment is the secret sauce that moves development from being project-based to business-as-usual; no need for a project plan, it’s just the job – come in, develop a feature, deploy it, go home. It looks like developers are now working with Ops (DevOps is everywhere) so closely that they are now testing and depoying their own code, and …. drum roll … it’s working! I have a train of thought going at the moment that says that if development costs moved from Capex to Opex (and innovation becomes a cost of doing business on the internet) companies would understand the requirements much better. No end date, no fixed scope – development just rolls on, day after day.

HTML5 was also hot this year – Google and Facebook were both touting their experiences / thoughts, and I wouldn’t be surprised if both didn’t settle on it as a core mobile development platform – which makes the demise of WebOS all the more poignant – right idea, wrong time?

The functional programming track was down from last year – server-side JS and Node.js in particular were this year’s favourites, with asynchronous programming the new concurrency paradigm. Since no one could claim production experience with Node, it’s hard to see where that’s going right now.

Overall, another great conference, good speakers, and some interesting experiences shared. There wasn’t a truly memorable talk, but that aside, as someone said (and I’ve said before) – we are in a golden age for software development – we may never have it so good again, so get out there and make the most of it.

QCon 2011 (London): Day Three

2011-03-13T23:41:00.001Z

Day three went a bit more webby – and technical, and frankly, a bit flat. It seemed slightly lacklustre.

Keynote: Things I Wish I’d Known (Rod Johnson) 7/10:

Co-founder of SpringSource, Rod is a successful entrepeneur, and entertaining speaker. He gave us a very personal take on the lessons learned, although I’m not sure I personally took anything away from it. I didn’t take any notes, so can’t really comment.

Single Page Apps and Future of History (Michael Mahemoff) 7/10:

Good talk from Michael (Googler) on the history of the web, and what single page apps will mean. Not much new here (for me) – this was more of a review session, but Michael put it all into context. A good discussion of the hashbang debate followed, along with things like REST, Hijax, and the history API of HTML5 (push/pop/replace) – I’m still not totally convinced by SPAs, but they do seem to be gaining ground. For me, the summary is that many apps these days are very task focussed, and so a SPA may fit, but for a larger site, pages are still the way to go.

I had actually been hoping for a session on WebSockets, but apparently that was cancelled. Que sera, sera.

Node.js: Asynchronous I/O For Fun and Profit (Stefan Tilkov) 7/10:

Node.js is a very hot topic right now, and this session was packed to the rafters. Stefan gave us a great run-through of the model, and some possible uses. This was a code-level demo – although it was noted that no one in the room (inc. Stefan) had experience of running Node in production. Definitely one to watch, and I’m glad I went, but it was more out of interest than a practical requirement.

HTML5 @ Facebook (David Recordon) 7/10:

Key take-away here was the use of HTML/CSS/JS for mobile application development – clearly a focus at Facebook. David called out PhoneGap specifically, which kind of torpedos any other “appcelerator” framework given that it has the Facebook seal, and is free.

There was a bit of a review as to how Facebook use HTML – specifically the BigPipe process – which is very clever, but not exactly news.

Basically – they’re really clever at Facebook, and have more money than anyone else. Copy them with care – it may not work out quite as well as you had hoped.

ThingWorx and Neo4j (Rick Bullotta & Emil Eifrem) 7/10:

Bit of a sales pitch from Rick around his company, ThingWorx, which I didn’t really get (database of everything, and how it’s related to everything else), with not enough really information on Neo4j, the graph database.

HBase @ Facebook (Kannan Muthukkaruppan) 6/10:

OK, so it was the last presentation, but it was also very, very technical. This was pretty serious stuff, and I lost track early on. As before, they’re cleverer than you are, at Facebook, so copy carefully.

QCon 2011 (London): Day Two

2011-03-13T23:03:00.001Z

Day two was (for me) primarily about the big guys – Google, Twitter, Netflix & Best Buy all made appearances, so this was less about the Special Forces of day one, and more about how that theory scales.

Keynote: Innovation at Google (Patrick Copeland) 8/10:

Great talk from Patrick Copeland on Google culture. Hard to summarise in short-form, but essentially the emphasis is on innovators (people) and not on ideas. Ideas are worthless without execution, everyone has an idea (Google themselves have 100-200k employee ideas in their idea-database), but very few people can make something out of an idea. Look for the spikey, difficult people – constantly challenging, never taking the easy route. Be prepared to fail, and fail fast – use data mercilessly to measure an idea, and if it’s not going anywhere kill it – Wave is the poster child for this philosophy. Google ‘dog-food’ anything and everything – not just new ideas per se, but even within the core product. Every search results page served is a complex multi-variate test (I’ve noticed this myself), measured and analysed. Like the previous day’s talks – no one should get too attached to anything – as everything is expendable. Fun section on the concept of the “pretendo-type” or “pretotype” – an example being the Palm guy who made a fake Palm Pilot out of wood and carried it around with him to see if it felt right. (See pretotype.org.)

One possible newsworthy item here was the impending launch of “Androgen” – a “v. fast prototyping tool for Android”. Watch this space.

Building Best Buy’s RESTful Commerce Engine (Brian Sletten) 8/10:

Starting to get into deep tech here, which was great. I worked with BBY a few years back, so have some appreciation of the complexity of creating software for such a behemoth, and they seem to have done a great job on their commerce API. The secret – well, there is no silver bullet (of course), but they seem to have taken a pragmatic approach – start simple, listen to your users and build out. I loved the purity of their implementation – particularly the use of hypermedia to provide fully-formed URLs to client software. You call the initial URL, and it will return all of the available service URLs. If they release a new service, they add it to the manifest, and you use it when you need to. Additional features included the use / extension of the link/rel attribute.

Our old friend “small teams, co-located” cropped up again – is there anyone who still believes that anonymous, distributed, teams of developer-drones can succeed?

One thing that they apparently found very useful is BDD – they have a very complete set (1,000+) of Cucumber tests that they can run – which apparently help enormously with smoke / regression testing new releases.

Data Architecture at Twitter Scale (Nick Kallen) 7/10:

Now we’re getting very deep – this was the first of a number of very technical presentations, which I can’t really do justice to here – though I would recommend the presentation when it finally appears on InfoQ. It was a really interesting data history lesson on Twitter – how they migrated from the original Rails / MySQL implementation to the (incredibly exotic) system they run today. Suffice to say, I now understand why the API restricts you to the last 3,200 tweets. They are fortunate in being to optimise for some incredibly precise scenarios, and so have built an architecture that supports those specific use cases. As one example, all of a user’s read-only timeline is cached in memory – so that when someone tweets all of their followers receive a message (pub-sub) that is then prepended to their own in-memory timeline. They have internal SLAs around every scenario – which includes propagating updates within a second – just imagine what that means when Ashton Kutcher tweets… (they can support 4.8m messages/second).

Couple of interesting bits: all engineering solutions are considered transient (following the theme), good enough is considered good enough, and until recently social graph events (following/unfollowing) exceeded tweet events. Counter-intuitive, but true.

Behind the Scenes at VISA (John Davies) 8/10:

This was a first – the only unrecorded session (I believe), this was a walk into the murky world of the world’s finance backbone. I hesitate to repeat anything here – I fear for my safety! John’s own presentation was neutered by the policy wonks at VISA, so this was largely a Q&A session. Interesting factoids: VISA runs on a mainframe (is not distributed), it runs on 99.99999% availability, it provides the US government with their GDP figures, it can afford to lose two data centres without interrupting service. No one knows where the DCs are, and their core systems have probably only changed 10% in twenty years (contrast with Dan North’s software half-life measured in months). This is the opposite of Agile, and probably good for it – this is, next to ICBM programming, about as serious as it gets. But that also presents VISA with a problem – when Hyves (Dutch Facebook) introduced a payment system, VISA card transactions dropped 25% in nine months – if Facecbook (/Google / ?) were to do the same, VISA’s entire business could collapse, and with their bunker-mentality they would have no way to react. So, they’re getting out into the community, hence John’s presence at QCon – VISA needs the development community more than we need them – but this could present some life-changing opportunites to some.

Be warned though – if you work at VISA you may have to sign quite a long, restrictive, covenant.

Netflix’s Cloud Data Architecture (Siddarth Anand) 8/10:

Some great facts re. Netflix – they were spending $600m/year on USPS postage before embracing the web, streaming costs 1% of posting a DVD, at peak they account for 20% of downstream bandwidth in the US, they have $2bn in revenue, and they do all this with 400 employees, 15 of which are in IT operations. That’s 15. Extraordinary stuff.

Anyway, they decided to move everything to AWS, which they did, in commendable JFDI style. One interesting point – they never attempted a wholesale data import from DC to cloud – but instead moved a user’s records on demand – when the user first accessed the new site. This meant that they were able to move everyone across over a two-three week period.

Yet again, as if it needed re-emphasising – small teams of capable people, empowered to do the job.

Using Hypermedia Services for System Integration (Tim Ewald) 6/10:

This was the end session of the REST track, and I felt a bit like I’d walked into a private conference. This talk was quite opinionated, and there was a lot of speaker-audience chat about how important it was to use link/rel for hypermedia, or some other document annotation. I don’t really care – the theory of hypermedia is interesting in general, and specific implementations are interesting (qv BBY session), but frankly the academic details are not. This was a classic case of over-thinking. A bit more JFDI, a little less ivory-tower please.

QCon 2011 (London): Day One

2011-03-13T20:53:00.001Z

Great opening day to QCon, with quite a consistent theme – we appear to have moved beyond Agile to a new post-Agile “lean development” model. This was echoed from the keynote through three of the five talks. This is the anti-offshore, anti-project-management, “let us get on with it” model – which may not work for many companies, but does appear to pay huge dividends to those brave enough to embrace it.

Keynote: Scaling Lean & Agile (Craig Larman) 8/10:
Keynote speech, interesting, insightful take on how to manage effective development across very large (1000+) teams. Essentially the QCon basics - hire great people, get out of their way. Avoid job titles - foster a culture of the master programmer - banish the word architect, and create cross-functional teams. Spend 5-10% of a sprint digging deeper into requirements for the next sprint. Great software engineers are gardners, not architects. Oh, and avoid project management tools where possible - it adds little except overhead.

From months to minutes - upping the stakes (Dan North) 8/10:
Taking it up a notch, Dan North (a QCon regular) spoke about his conversion from Agile to the new post-Agile world - working in financial trading firm where software is released every three seconds on average. Or something like that. Dan's a good speaker, giving an entertaining talk, and a lot of what he said resonated with the keynote - small teams, tightly integrated (inc. the 'client'), with a lot of autonomy and control. You would have to build up a heck of a lot of trust to get away with this! Most interesting concept he spoke about is that of "Software Half-life" - this time taken for 50% of the codebase to be re-written. He estimated theirs in months - the codebase is constantly refactored out of existence; in a world like this, no one gets hung up on their own code perfection - get it done, and out of the door ("spike-and-stabilise"). He also spoke of the "path-to-production" being the most effective test-bed. True, but again, you'd need an awful lot of trust on account to be able to live like this. Fun though.

Lean Programmer Anarchy (Fred George) 8/10:
Taking it up another notch (or ten), Fred George introduced us to "Developer Driven Development" - a world without QA, BA or PM - just developers and the client (internal). This is a world where developers are measured on their contribution to the bottom line - so frankly no one gives a damn about the backlog or what you did yesterday. Hard to believe it really works like this - but apparently it does, and the business now thinks that more developers = more profit, so on they come. Developers hire their own, self-organise and manage their own time. Extraordinary, but a great fun presentation. One interesting point that reflected the keynote was the "cult of the programmer" - a/c Fred, in the bad old days (of waterfall), being a developer was a crappy job (true), and so any decent developer wanted to move on to becoming an "architect". This is no longer the case - as Fred put it - how many surgeons wake up one morning and decide to become a hospital administrator? Developers can now grow old at the keyboard, like geeky rock stars, which is a great thing (IMO).

Bringing Developers and Testers Together with Visual Studio (Giles Davies) 4/10:
Talk about coming down to earth with a bump! I walked into this talk by mistake - a last minute room-change meant that the "Do's and Don'ts on Android" talk I was expecting was in the adjacent room. In contrast to the previous talks (and the keynote) - the word from MSFT was "do you want developers having access to the test environment - probably not!" - cue tumbleweed. This was so at odds with the tone of the day (developer anarchy = profit) that it was hard to imagine it was the same conference. I felt sorry for the presenter - it wasn't his fault, and he did a grand job of demonstrating some convoluted Visual Studio features - I'll give him 4/5 for presenting and the content 0/5. Topped off with some Q&A around licensing. Wrong conference MSFT.

Agile Operations (Dan North and Chris Read) 8/10:
Dan teamed up with (another ex-ThoughtWorker) Chris Read here to demonstrate how it all comes together. Great fun, and a repeat of a (ThoughtWorks) presentation from QCon SF that I watched a while back - basically reinforcing the concept of "Infrastructure as Code" - a lot of which depends on tools like Chef / Puppet / CFEngine etc. Putting everything into version control, rolling out changes to infrastructure like you would with code. The advantage of versioning includes having a comment attached to each change - no more mystery DNS updates! NB - these guys still deploy application code using hand-rolled shell scripts - the Chefification is for the network infrastructure only. Echoed one of the other themes of the day - "optimise for change" - change will happen, so embrace it and build it into your work. Like Etsy - these guys often find it's quicker to redeploy than rollback - sign of a good process IMO.

Making Apps That Don't Suck (Mike Lee) 5/10:
Mike Lee is a legend, apparently. He's built some great apps on iOS, made some money (I guess), and is now travelling around the world dressed as a pirate. He's also a bit of a tit - and whilst I was interested in his views on iOS development, something he clearly knows about, I wasn't remotely interested in his views on life - which were of fortune-cookie sophistication. I have subsequently read that he was fired (http://techcrunch.com/2008/08/26/tapulous-cofounder-mike-lee-ejected-from-company/) from his own company. No surprise to me.

NASA Keynote (Mark Powell) 6/10:
Second keynote of the day - this from a NASA guy, although not, much to the audience's dismay, someone writing software for spaceships. Still, it was interesting stuff - they clearly have some clever people working at NASA, but generally a downbeat end to the day.

Shoreditch – partying like it’s 1999

2010-12-08T10:50:00.001Z

I’m a bit late on this one, but when I read about this on holiday - http://www.wired.co.uk/news/archive/2010-11/04/david-cameron-silicon-roundabout – I thought I was dreaming. Silicon roundabout has apparently become mainstream, but this is very, very old news (even in real, i.e. not-internet, time).

I went to my first Old Street rooftop digital / web collective launch party in 1999 – Hoxton Square was the place during the original dotcom bubble. Lastminute.com was headquartered down the road in Farringdon, I went to an interview at Gorrilla Park Ventures in a converted warehouse near Spitalfields – the idea that the tech community has suddenly converged on the strip running from Farringdon through Old Street to Shoreditch is pure marketing fluff.

Perhaps we’re just waking up after the 10-year hangover.

Web Forms (1): Validation

2010-11-05T12:54:00.001Z

This post is more of a note-to-self than anything. Part one of a two-parter, concentrating on web form processing – first from a server-side point of view, and then later, from a user-centric client-side point of view. The aim is to document a pattern for forms processing that is consistent and repeatable for any website. Think PRG (Post-Redirect-Get), with some server-side detail added in for good measure. PRG+.

In this post, I just want to focus on server-side data validation, and how to deal with the different types of invalid data entry. The first point to note is that you should never make any assumption about the origin of the data. You must ignore any notion of client-side validation – the data that your controller receives as the request could have come from anywhere, and may contain anything. Data is sent over the wire as text key-value pairs, and should therefore go through a number of validation steps:

Checking the data type – everything arrives as a string over the wire, so needs checking
Simple data type logic – is an email in a valid format, is a date of birth set in the past
Domain logic – is an email / username a duplicate, does a product exist, is the price ‘right’?

Only at this point can you attempt to process the form with any confidence.

When an rule fails, there are a couple of different ways in which the user can be notified:

Return to the form, and allow them to amend the invalid data
Proceed to a new page, with appropriate messaging (e.g. PRG)

I will go into more detail around this side of the process in the next post (when to redirect, how to notify users etc.), however one thing I would highlight here is the difference between recoverable and unrecoverable errors: if the data fails validation before any changes are made server-side, then I believe the user should be alerted via a warning message and not an error message. Errors should indicate that something went wrong, and if a form fails validation then the user should always have the option to amend their data and resubmit it. In project management terminology, warnings indicate a risk (something that could go wrong if mitigating action is not taken), whilst errors indicate an issue – something that has gone wrong already.

I also believe that there is a difference between the simple data validation and the contextual business domain validation, and that only the first type should ever be replicated client-side (ignoring AJAX for now – more of that next time). In an MVC world, I think the first type can be done by the controller, and that this validation should match client-side validation, whilst the second type should be done deeper into the model and / or application domain. (Assuming that the controller is just that – a controller – and that it delegates the ‘doing’ to other components.)

Below is some pseudo-code demonstrating what I believe** to be the ideal processing and validation for a sample form request handler.

ProcessForm(request)
{
    /* first do some 'dumb' data type validation of input values – remember
    that all request values are passed over the wire as text, so they need to
    be validated according to the basic destination data type. At this point
    client-side validation should be ignored – we don’t know that the information
    was submitted using the form – we simply know which controller was called. */
    if (! isEmail(request.email))
        view-data.errors.add(new InvalidPropertyException("email"))
    if (! isZipcode(request.zip))
        view-data.errors.add(new InvalidPropertyException("zipcode"))
    if (! isDate(request.dob))
        view-data.errors.add(new InvalidPropertyException("dob"))
    etc.

    /* if we have any errors so far then don't bother continuing, return to the
    form and prompt the user for new values. This is WARNING, and not an ERROR,
    as nothing has been changed server-side, and the user can always amend the
    values and resubmit. This is the server-side equivalent of client-side JS
    validation. */
    if (view-data.hasErrors)
        returnToFormAndHighlightIssues()

    /* if we get here then we know that the values are 'correct' but that doesn't
    mean they will work. This next validation step may require more context than
    the simple validation above – it is not something that can (or should) be
    replicated in client-side JS. e.g. is the price in the acceptable range, is the
    username available. */
    try
    {
        model = new model(request.email, request.zip, request.dob, request.price)
        doSomethingWithModel(model)
    }

    /* exception thrown by any property setter that doesn't like the value it's
    given; this is functionally equivalent to the case above - nothing has really
    happened so log this as a WARNING, and not an ERROR */
    catch (InvalidPropertyException)
    {
        view-data.errors.add(theException)
        returnToFormAndHighlightIssues()
    }

    /* exception thrown by the doSomethingWithModel method that occurs BEFORE
    anything has been committed, and whilst there is the opportunity to resubmit
    the data. An example of this might be an attempt to register a duplicate
    username; this can be marked as a WARNING or an ERROR, depending on context.*/
    catch (RecoverableException)
    {
        view-data.errors.add(theException)
        returnToFormAndHighlightErrors()
    }

    /* exception thrown by the doSomethingWithModel method after data has been
    irreversibly committed. In this case resubmitting the data is not desirable,
    and the user should be alerted! An example of this might be a database
    exception after the record has been partially committed. */
    catch (UnrecoverableException, AnyOtherUnexpectedException)
    {
        view-data.errors.add(theException)
        renderDifferentPageWithMoreInformation()
    }

// phew - if we got here it all went well, so we render the anticipated page
renderExpectedPage()
}

** It would be fair to say that my views on this aren’t universally accepted – comments welcome.

Guardian Open Platform – IT, or Innovation?

2010-10-20T00:45:00.001+01:00

Earlier this week I attended the Enterprise Search Meetup that was held at the Guardian’s office, and entitled “Search at the Guardian”.

It was a great event, lots of interesting details and a good insight into the Guardian, who, through their Open Platform initiative, seem to running away from the rest of Fleet Street in terms of innovation and open data. Who knows, perhaps they even “get” online?

Plenty of others have written reviews of the event, and most of the speakers themselves blog regularly about the Open Platform, so I won’t go into that here – suffice to say that, in the words of Martin Belam, “Open Data is basically one big advanced search query”.

What I found most interesting about the evening was the philosophy / approach behind the initiative, something that I believe is neatly illustrated by their choice of search software.

The Guardian held a search “bake-off” when they had to decide whether to upgrade their Endeca implementation to a new version, and they invited Endeca, FAST and Apache Solr. The eventual winner, and apparently this was a decision that ignored commercial factors, was the ‘free’ one – Solr. And why – well, because when it came down to it, Solr was more developer-friendly, and they knew that when they started to push the boundaries of the product, they could either reach out to the community, or, heaven-forbid, do it themselves.

This is not a rant about Open Source Software, however, as I’m not sure that’s the point. I think the Guardian is probably spending more on people tweaking their free software by hand than they would have spent on the licence fee for the expensive commercial software equivalents.

To me, the point is this: on the plus side, the commercial product comes with a supportable SLA, a sense of security around the product vendor, and a contract, which allows the user to hold the vendor to account in the event of failure. It also comes with a set of documented tools and user manuals – a product is designed to slot into an operational model.

On the downside, it is very hard to innovate around a commercial product – if it doesn’t do what you want it to, you simply have to sit and wait for the next release.

To an IT-style "CIO”, this is perfect – a fixed contract allows them to manage costs and budget for the year ahead – as well as safe-guarding their position (it would be hard to fire someone for choosing to buying Endeca over Solr in a large organisation.) It reinforces the idea of technology as an operational cost, to be managed down. This is is the essence of “IT”.

The alternative approach is to invest that same money in people – clever people, empowered to innovate. For the same amount of money they are now treating technology as a competitive advantage, something that can be used to put clear water between themselves and their peers. It’s a huge gamble, and the person responsible almost certainly will lose their job if it doesn’t pay off.

In answer to my question on the night - “What is the commercial model, and why would anyone pay to support this initiative”, it was clear that whilst there is no direct commercial model as such (it’s free, after all) they were already seeing benefits. As Stephen Dunn (whose decision it may have been) pointed out, by opening up their platform they are capable of supporting far more partners than they would have been by trying to integrate them into their core systems. (I’m not sure this was the original design goal, but I may be wrong.)

A brave initiative from the Guardian, and one worth supporting.

DNS Propagation

2010-10-19T23:33:00.001+01:00

We all tell clients that DNS changes take 24-48 hours to propagate, but we very rarely get to see any evidence of this. Well, now I have a nice chart showing just this.

A website that I am, ahem, involved with, went offline for a period of time earlier this week on account of a ‘miscommunication’ with the company hosting the DNS name servers. As a result, they removed the DNS record from their servers, and the domain simply disappeared from the internet. The solution was pretty simple – update the name servers to the current hosting provider (the ever reliable Rackspace, who I have no hesitation in plugging – their service desk is fantastic). Then all I had to do was sit back and wait.

Fortunately, at the same time the site was being monitored by the equally reliable Pingdom, who monitor the site from 25 locations in 10 countries across Europe and North America (not representative of the world, I know, but enough to make this interesting).

This meant that I could watch the propagation of the DNS change as it was picked up by their servers. Again, this is not a scientific survey, but interesting nonetheless.

User-centred design

2010-09-30T21:53:00.001+01:00

In a follow-up to my previous post on admin interfaces, I thought I’d post on something that happened the other day on the current project.

As part of the ecommerce site we’re building we have a dedicated service for managing email notifications (registration, order confirmation, lost password etc. – about 50 in total), and during development we’ve been using Google’s SMTP service to do the sending itself. A couple of days ago I wanted to find out whether an email had been sent (as it hadn’t been received), and the easiest way to do that was to log in to the associated Gmail account and look into the Sent Items folder.

Turns out, the best logging / admin tool for an email service is, well, an email service. You can search through the emails, filter them, track them, configure various options and more. It just works.

At the same time, whilst thinking through the process for website administrators to take down certain products, the solution is not to give them access to a special website from where they can use the the Take-Down function (for which they will need the appropriate training and documentation), it’s to put a Take-Down button on the website product page that is only visible to people who are administrators. No “process re-engineering” required.

This user-centred design concept may yet catch on…

(Of course, it can get a bit out of hand – qv the “customer journey re-engineering manager” comment in this article - http://hrb.fm/aMakDY – rather embarrassingly I think I know the people involved!)

HTTP self-service

2010-09-26T20:02:00.001+01:00

HTTP is all around us – it has become the de facto transport protocol of choice; web servers may be the primary source, but routers / access points have used it for a long time, databases have begun to use it (e.g. CouchDB, RavenDB), and now even console apps have taken to its user-friendly interface. We’ve been using Mercurial on our latest project, which is a fantastic application in its own right, and despite being essentially a console application, it too contains its own HTTP server. Type in “hg serve” at the command line, and it will start up a single-use dedicated HTTP server, which provides a great online user interface.

I won’t go into the details (you can try it yourself), but it did set me thinking. Running background applications on a server have always suffered from the lack of user interface, but embedding an HTTP server in the application would allow you to offer this**. Moreover, within a locked-down production environment you could provide network access to an application using a protocol that sysadmins understand.

Digging a little deeper, I came across Kayak – a lightweight .NET HTTP server, which would allow just this. It should be possible to use this (or an equivalent) to bundle an HTTP server in with your services to serve up things like: logs, service status (e.g. databases up, network locations running) & configuration information. This would also allow you to manage / update configuration settings in runtime, and to combine multiple services into a simple management console.

Ultimately, it would be great to provide a mechanism for uploading software updates – in the same way that you update router firmware – navigate to a URL, upload an update package, and then restart the service. Add in scripting support, and it would make management of a multi-server environment much, much simpler, without the need for an expensive System Centre type application.

I’m so convinced that this is a good idea that I’m going to set up a project to demonstrate its application, so watch this space.

** NB – when I say “user” interface, I am obviously talking about the tech-team, not end-users. I’m not advocating opening this kind of interface to the internet – this is internal only!

Whatever happened to XSLT?

2010-09-13T18:28:00.001+01:00

Many, many years ago I built a website that used XHTML/XSLT to render all of the pages (this was back in the old-school ASP days). It worked a treat, and I’ve had a certain regard for XSLT ever since.

Which is why I was curious when some colleagues were tasked with building an email template system for an ecommerce system (large ecommerce systems can have dozens of email templates covering things like order confirmation, registration, forgotten password etc.) and then decided to use their own custom templating language.

Surely this is what XSLT was invented for – taking one block of XML (e.g. order details) and transforming it into another (an XHTML representation of the order)? “Too hard” came the reply; “it’s another language to learn” was another.

I wondered if this was a wide-spread complaint, so did a quick Google, and it appears as if XSLT has dropped into a black hole. It’s either so ubiquitous that no one talks about it anymore (it just works), or people are genuinely turned off by it.

It’s a shame, because with all of the fuss about DSLs these days, XSLT deserves an honourable mention.

Google’s “High Performance Image Serving”

2010-08-22T21:24:00.003+01:00

[Update] I stand corrected - I do now have billing enabled on my Apps account, and can confirm that images are served with all the correct response headers set. The URLs do indeed support 304 conditional GETs on the production infrastructure, which makes this a very attractive image hosting solution. Well done Google, apologies for the mis-representation.

(NB – this article is based on the development SDK (1.3.6) – I haven’t been able to test on the production infrastructure as yet as for some reason Google won’t authorize billing on my AppEngine account, without which the Blobstore is unavailable!)

The concept of a hosted service that manages image serving / caching, resizing & cropping is something that has cropped up in projects I’ve been involved with for the past 5 years (and the rest).

I had a hand in a startup about 6 years ago now that processed MMS (picture messages) that people sent in. At its simplest this involved storing people’s uploaded images, and then cropping / rotating and resizing the images to fit a certain profile (we were printing them out, and needed to convert landscape to portrait, and crop to a fixed aspect ratio.)

Subsequent to that experience I worked for many years in online digital entertainment, developing a platform for the processing of music / video assets, and one of the elements we also needed was the ability to take in hi-res images, shrink to an acceptable size / format (e.g. not TIFF), and then host them for delivery across a CDN.

The project I’m currently working on has a combination of the two – processing user-generated content for serving back over the web. What we need is an image processing service, backed with a large data store, and a high-performance cache.

So, it was with great excitement that I noticed that the Google App Engine SDK comes with an in-built library to do exactly this. It’s built on top of the Picasa library (it even includes the “I feel lucky” transformation), and enables cropping, resizing, rotation etc. The App Engine platform has no file-backed storage, but the datastore does include the BlobProperty type, which can be used to store binary data (such as images). A simple image processor using this took about fifteen minutes to set up (which was mainly cut-and-paste from their sample app here).

(Some people may by now be thinking “Tinysrc” – the online service that resizes images for mobile screens – well, no surprises, tinysrc runs on AppEngine – this is precisely how they do it, except that they pull the images from a remote server – they are not stored.)

The downside of this approach is the the datastore has a 1MB limit per entity, which makes it borderline useful if you’re dealing with UGC (web-optimised images should never be 1MB, but the image someone just uploaded from their new digital camera could easily be.)

Fortunately, Google provides a secondary datastore specifically for large binary objects, called the Blobstore. It’s well documented (here), so I won’t repeat that, but what I can say is that it integrates directly with the Image api (see here). There are some complex limits about the amount of data you can process, so read the article carefully, but suffice to say it can be done. (See here for a nice example of the blobstore / image api interaction.)

A killer function, which has been publicised this last week (as “High Performance Image Serving”), is the “get_serving_url” function in the Images api – which takes in a Blobstore object key, and returns a fixed URL that can be used as the static image URL.This looks almost like a Google CDN – the ability to serve images as static content, with the ability to crop / resize on the fly (albeit using fixed sizes) thrown in for free.

[Updated – see intro] And yet… if you set up an image service using these amazing (and practically free) resources, you’ll find a fairly large hole in the implementation. It’s our old friend HTTP status codes. The fixed URL exposed by the images service does not support a 304 (content unmodified) status – meaning that every time you call for it, you get the whole thing, increasing server bandwidth and client download times. (See introductory note – this may just be a development server issue – TBC.)

~~I can only assume that this is deliberate – as Google gets its money from the bandwidth charge. It is however extremely annoying.~~

Links:

Google: Blobstore API
Google: Image API
Google: App Engine blog article
Tinysrc: Image resizing service based on App Engine
Tin Isles Blog: sample code using get_serving_url