Planet TW

Sam Newman: Currently experiencing the guilty pleasure that is watching The Waterboy. No, it isn’t Punch...

Sun, 05 Jul 2009 21:46:37 +0000

Currently experiencing the guilty pleasure that is watching The Waterboy. No, it isn’t Punch Drunk Love but Kathy Bates rocks

Sam Newman: Given the leading topic, one suspects we may be seeing the end of the trending topics. Crowd...

Sun, 05 Jul 2009 21:46:37 +0000

Given the leading topic, one suspects we may be seeing the end of the trending topics. Crowd sourcing only works when crowds are smart.

Sam Newman: I’m officially in love with the shutter sound of the nikon fm3a. Now all I need are two more...

Sun, 05 Jul 2009 20:26:12 +0000

I’m officially in love with the shutter sound of the nikon fm3a. Now all I need are two more fast primes…

John Hume: AR⊗TA

noreply@blogger.com (John Hume) — Sun, 05 Jul 2009 17:33:33 +0000

If you haven't already, you should check out Brian Marick's Agile reboot: Artisanal Retro-Futurism Crossed With Team Scale Anarcho-Syndicalism.

There's a video of his excellent presentation from Agile Roots if you don't feel like reading. There are also stickers you can ask for if you feel like spreading the word.

It's totally tangential to the point, but my favorite quote from the video is near the end of the questions: "...in my observation agile teams care more about delivering business value, often, than the business they're delivering it to."

Felix Leipold: Seaside Event at the London Office this Monday

Sun, 05 Jul 2009 15:52:53 +0000

If you are in London this Monday don’t miss out on our Seaside themed GeekNight at 7pm.

Seaside is a truly revolutionary web framework implemented in smalltalk. We have Lukas Renggli representing the small but rapidly growing seaside and smalltalk open source community as well as Michel Bany of Cincom talking about his commercial experience with Seaside and Smalltalk.

It’s at the ThoughtWorks London office.

For more information about seaside and smalltalk have a look at the links in my earlier post.

Mark Needham: Brownfield Application Development in .NET: Book Review

Sun, 05 Jul 2009 14:43:40 +0000

The Book

Brownfield Application Development in .NET by Kyle Baley and Donald Belcham

The Review

I asked to be sent this book to review by Manning as I was quite intrigued to see how well it would complement Michael Feather's Working Effectively with Legacy Code, the other book I'm aware of which covers approaches to dealing with non green field applications.

What did I learn?

The authors provide a brief description of the two different approaches to unit testing - state based and behaviour based - I'm currently in favour of the latter approach and Martin Fowler has a well known article which covers pretty much anything you'd want to know about this topic area.
I really like the section of the book which talks about 'Zero Defect Count', whereby the highest priority should be to fix any defects that are found in work done previously rather than racing ahead onto the next new piece of functionality:

Developers are geared towards driving to work on, and complete, new features and tasks. The result is that defect resolution subconsciously takes a back seat in a developer’s mind.

I think this is quite difficult to achieve when the team is getting pressure to complete new features but then again it will take longer to fix defects if we leave them until later since we need to regain the context around them which is more fresh in our mind the earlier we fix them.
Another cool idea is that of time boxing efforts at fixing technical debt in the code base - that way we spend a certain amount of time fixing one area and when the time's up we stop. I think this will work well as an approach as often when trying to fix code we can either get into the mindset of not fixing anything at all because it will take too long to do so or ending up shaving the yak in an attempt to fix a particularly problematic area of code.
I like the definition of abstraction that the authors give:

From the perspective of object- oriented programming, it is the method in which we simplify a complex “thing”, like an object, a set of objects, or a set of services.

I often end up over complicating code in an attempt to create 'abstractions' but by this definition I'm not really abstracting since I'm not simplifying but complicating! This seems like a useful definition to keep in mind when looking to make changes to code.
Maintainability of code is something which is seriously undervalued - I think it's very important to write your code in such a way that the next person who works with it can actually understand what's going on. The authors have a fantastic quote from Perl Best Practices:

Always code as if the guy who ends up maintaining your code is a violent psychopath who knows where you live.

Writing code that is easy for the next person to understand is much harder than I would expect it to be although on teams which pair programmed frequently I've found the code easier to understand. I recently read a blog post by Jaibeer Malik where he claims that it is harder to read code than to write code which I think is certainly true in some cases.
There is a discussion of some of the design patterns and whether or not we should explicitly call out their use in our code, the suggestion being that we should only do so if it makes our intent clearer.
While describing out how to refactor some code to loosen its dependencies it's pointed out that when the responsibilities of a class are a bit fuzzy the name of the class will probably be quite fuzzy too - it seems like this would server as quite a useful indicator for refactoring code to the single responsibility principle. The authors also suggest trying not to append the suffix 'Service' to classes since it tends to be a very overloaded term and a lot of the time doesn't add much value to our code.
It is constantly pointed out how important it is to do refactoring in small steps so that we don't break the rest of our code and to allow us to get rapid feedback on whether the refactoring is actually working or not. This is something that we've practiced in coding dojos and Kent mentions it as being one of his tools when dealing with code - I've certainly found that the overall time is much less when doing small step refactorings than trying to do everything in one go.
I'm quite interested in trying out an idea called 'Bowling Scorecards' which my former colleague Bernardo Heynemann wrote about - the idea to have a card which has a certain number of squares, each square reprsenting a task that needs to be done. These are then crossed off as members of the team do them.
An interesting point which is made when talking about how to refactor data access code is to try and make sure that we are getting all the data from a single entry point - this is something which I noticed on a recent project where we were cluttering the controller with two calls to different repositories to retrieve some data when it probably could have been encapsulated into a single call.
Although they are talking specifically about poor encapsulation in data access layers, I think the following section about this applies to anywhere in our code base where we expose the inner workings of classes by failing to encapsulate properly:

Poor encapsulation will lead to the code changes requiring what is known as the Shotgun Effect. Instead of being able to make one change, the code will require you to make changes in a number of scattered places, similar to how the pellets of a shotgun hit a target. The cost of performing this type of change quickly becomes prohibitive and you will see developers pushing to not have to make changes where this will occur.
The creation of an anti corruption layer to shield us from 3rd party dependency changes is suggested and I think this is absolutely vital otherwise whenever there is a change in the 3rd party code our code breaks all over the place. The authors also adeptly point out:

The reality is that when you rely on another company's web service, you are ultimately at their mercy. It's the nature of third-party dependencies. You don't have control over them.

Even if we do recognise that we are completely reliant on a 3rd party service for our model I think there is still a need for an anti corruption layer even if it is very thin to protect us from changes.

The authors also describe run time and compile time 3rd party dependencies - I think it's preferable if we can have compile time dependencies since this gives us much quicker feedback and this is an approach we used on a recent project I worked on by making use of generated classes to interact with a SOAP service rather than using WCF message attributes which only provided us feedback at runtime.

In Summary

This book starts off with the very basics of any software development project covering things such as version control, continuous integration servers, automated testing and so on but it gets into some quite interesting areas later on which I think are applicable to any project and not necessarily just 'brownfield' ones.

There is a lot of useful advice about making use of abstractions to protect the code against change both from internal and external dependencies and I particularly like the fact that the are code examples showing the progression of the code through each of the refactoring ideas suggested by the authors.

Definitely worth reading although if you've been working on any type of agile projects then you're probably better off skim reading the first half of the book but paying more attention to the second half.

Sumeet Moghe: The Agile Elearning Design Manual - Of Project Spaces & Project Project Managers

noreply@blogger.com (Sumeet Moghe) — Sun, 05 Jul 2009 14:32:34 +0000

Whether you're part of a consultancy or an in-house training outfit, you're providing a service to a customer your people should matter a great deal to you. A few years back, I read Guy Kawasaki's Art of the Start and Guy insists that product manufacturers and service providers should niche themselves. He explains the thought using a graph like the one you see above. We measure a product/ service by the value it provides to the customer and its uniqueness. A highly valuable but non-unique service competes only on price, because there are others providing it too. You're perhaps stupid if you provide a unique service and no one wants it. You're perhaps a dotcom if you're providing a non-unique service which no one wants. All this said, there's no arguing that your position is most enviable if you provide a service that's unique and highly valuable to the customer. As internal or market service providers, tools and methodologies don't make us unique, because others are only a step away from adopting them. What makes us unique are the people that deliver it.

A few years back, Martin Fowler our chief scientist wrote that People Matter Most in software development and one of the reasons he joined ThoughtWorks was because he saw ThoughtWorks as Roy - our founder's social experiment. In many ways, as a software delivery and technology consulting firm, what makes ThoughtWorks unique is the outstanding ability of ThoughtWorkers. Its a place I'm quite proud and privileged to be part of. As I've mentioned earlier, elearning development has got extremely commoditized and should really not be very different from software development. Its really important then, that we give people as much importance in elearning development as we do in software development.

Whether you run an elearning consultancy or are a training manager or a team member on an elearning development team, its important to think of how to empower people on your team. How much is your organization maximizing their ability? A lot of what I'm about to suggest in this article isn't necessarily a trivial change. Having said that, if you find value in implementing any of the suggestions I'm making, you may need to plan this change and transformation for the benefit of your team.

Growing Versatilists on the team

We saw in an earlier post, how there's very little overlap between the skills of different people on a traditional elearning team. I've revisited the current state of the skill map on most elearning teams I've seen in the recent past and right alongside, is what I think should be the minimum overlap of skills in the team. A Instructional writer/ designer should be able to do some testing and some building work; a builder should be able to do some writing, some testing and some graphics work and a graphic artist should understand the basics of building. Of course, any versatility beyond this helps. I'm a big advocate of Learning Generalists as against Specialists, so the more skilled a professional, the higher they rate in my book.

How environment affects versatility

The above floor layout will create a sense of deja-vu for those who've been in my position. This is a floor layout I observed first hand, four months back at an elearning firm. The seats I've pointed out are those of people on the same project. The graphic artist being a freelancer, doesn't even work on the same floor. I find it funny to think that people communicate using documents to subvert this kind of a layout. I can understand the limitations of forced distribution, but really that isn't even the case with most elearning teams. Its not hard to imagine why a lot of people stay specialists in just one trade. Its not surprising either, to understand why there are so many silos in some elearning outfits!

The waste of documentation inventory

The above picture represents the typical flow of work across silos. Notice the amount of documentation that gets created in this process. The learning consultant has a "consultanty" conversation with the customer to determine the "Instruction Strategy". This goes into a document that gets passed a few desks across to the Instructional Designer. The designer spends the next few weeks writing out a script, filling out matrices and creating more documentation to pass on to the builder. The builder on the other hand passes on graphics descriptions in a document to the graphic artist. The graphic artist sends out images based on his understanding of the document and if we're lucky then the back and forth isn't very long. The builder then passes on the completed module to a tester and the tester sends back documents with bug descriptions to the builder. If we're lucky then this back and forth isn't very long. All this is usually governed by a set of technical specifications documented by the Technical Consultant who sits nowhere near anyone doing this work!

Now I'm not trying to say that documentation is bad! What I'm trying to say is that documentation as a substitute for face to face communication is not ideal and is really a waste. The biggest waste in this process is the elearning script. Why do we need to write a script for 200 screens of elearning, when:

all that the builder can build in a day is say, 15 screens;
we should be showing the client training for the highest priority actions first;
we can simply explain most of these screens through face to face conversation;
an initial showcase of our work could render all of the scripting useless!

Simplify your work environment

A small, yet highly political change of the workplace can hugely simplify this approach. What if everyone sat on one table with no cubicles? People can choose seating that was convenient for the day -- depending on who they need to collaborate with. We can shout out ideas across the table since we're all within earshot. All of a sudden writing documents doesn't even seem like an option. Its so much easier to talk to the other person. Yes I understand that "Learning Consultants" and "Technical Consultants" can't always sit at the table because they have their fingers in many different projects. Its important to think of whether you need these roles in the first place. Given that your overall design is simple and modular, most projects can do without these overheads. If you do need someone like them though, they should be obliged to spend some time on a regular basis with each team.

De-managing your project - think "Collective Ownership"

I strongly believe that every group initiative needs a leader. I'm not so sure about layered management though. Its interesting to see how you can put a bunch of smart people together and see how they take charge of a situation once you allow them to self organize. This is a huge paradigm shift from the command and control structure that I've seen in some elearning outfits, but then I'm fast seeing other outfits that believe more in collaboration than coordination. There are a few management questions I still need to answer, so let me try my hand.

Managing Scheduling

Now if you noticed the project manager was doing a lot of scheduling work to get the right person to work on the project at the right time. This is no easy task -- almost equivalent to mastering the space and time continuum! Often, there's either a person waiting for work or work waiting for a person. This idle time is a waste in the process. A simple design up front, colocated-team-on-a-table approach takes away the scheduling overhead from the project manager, because every person downstream in the process pulls work from the person upstream. Which is to say:

We keep design simple, by using an action mapping approach;
We ask the customer to prioritize actions based on their perception of business value;
We (builders and designers) storyboard the most important action and activities first;
The builder and designer work to build the first activity together;
The graphic artist builds the visuals in collaboration with the builder;
The tester tests the first set of screens;
The customer looks through this set and signs off;
In the mean time, the process has already started another cycle with another activity

Try a Card Wall

The traditional project manager, who would be obsessed with a project plan can now create collective ownership in the team by using a Card Wall. A card wall is a simple visual representation of the work in progress for the current iteration. Depending on the different stages of work that each card goes through, you can construct the wall by making swim lanes for each stage. The progress of an individual card is indicated by its place on card. With a card wall, its quite easy to see where your bottlenecks are. For example, if you see a lot of cards in the "Waiting for Testing" lane, then it means that your testers perhaps need help in clearing out activities/screens waiting for testing.

On Agile teams, each morning the team meets for 15 minutes for a standup, describing what they did the previous day, what they're doing today and their blockers. This discussion happens in front of the card wall, so that issues can be pointed out in correlation to the work in progress. This empowers the team to find solutions to their own problems and generally creates a great problem solving culture in the team. With a colocated SME/ customer you can avoid creating reports as well (if they don't mind). The card wall is a snapshot of the work in progress for the team. If you need more convincing, read this post to see how such a lightweight tool makes a huge difference.Now I fully agree that this is difficult for a distributed team. I strongly recommend the use of a collaborative project management tool such as Mingle, to facilitate this kind of a process. I'll try to address that in a future post.

So, if the team is empowered to pick up work on their own, manage their workflow and solve their problems, what need do we have for a project manager? Here's where I feel the project manager could play a project leader. I've written earlier about what I believe to be the tenets of leadership. The project manager/ leader's time should get spent in coaching, mentoring, absorbing external pressures and providing help and guidance in removing team blockers. In time, the project manager builds these essential leadership skills and by helping to remove blockers for the team, can possibly emerge as the team's ultimate generalist.

How did you like this post? Please feel free to let me know. If you liked this post, you may just like some of these other, related posts:

The Agile Elearning Design Manual - Problems with existing Approaches

The Agile Elearning Design Manual - Agile Re-explained

The Agile Elearning Design Manual - Think Small (Iterations, Action Maps, Storyboards, and Mini-Modules

The Agile Elearning Design Manual - Iterations huh?

My Tenets of Leadership

From Training Specialist to Learning Generalist

Why your Training Team needs Versatilists

Luca Grulla: What has to be ready for the beginning of a project ?

Sun, 05 Jul 2009 13:08:08 +0000

The beginning of a project is always a hectic period where several things have to be put in place in order to be able to start the actual development from a solid position.

Interestingly enough, I see that what is important to have ready for ThoughtWorkers most of the time is not what has to be ready for other people.

Given that obviously every project is different and deserve specific attention, here is my list of things that has to be ready before the kick off of iteration 1.

Infrastructure

A repository, a Continuos Integration Environment with Cruise (or suitable alternatives) installed and working, a QA environment where we can deploy every successful build whenever we want, a basic build script (i.e. build/run tests/package). Pairing boxes, each one with exactly the same configuration.

Architecture

Just identify the core services/components, there’s no need to go into detail for each one at this time of the project. If we identify what is the main responsibility of a service is good enough for now, the details will be discovered later in the project. If we can easily identify the way services will communicate (web service ? message broker ?) good, if not through good OO principles we can abstract the low level mechanism and reduce the cost of change later, deferring the final decision to the point in the project where we have more understanding of the technological constraints.

Patterns

Pairing and frequent pair rotation will help in spreading the knowledge of the approach to solve specific problems and in maintaining consistency throughout the code base, so most of the time there is no need of defining a sort of “project dictionary” of valid patterns at day 1.

If we’re introducing new approaches to solve a specific problem, it’s important to highlight pros and cons of the approach so that people know what they are doing once pairing.

Most of the time these are probably enough to start Iteration 1.

All the other decisions can (and sometimes should…) be deferred to a later stage.

Ye Zheng: 乐乐呵呵写程序

Sun, 05 Jul 2009 10:42:19 +0000

很多人把软件开发形容得太辛苦，为了解决一个问题，需要耗尽心力和体力，更有甚者，Ed Yourdon把软件开发说成了“死亡之旅”。

很多到我们公司参观的人，都会注意到，我们办公桌有着各种各样的小玩意，通常它们扮演的角色就是Standup中发言的Token。当然，我们互相打闹的时侯，它们就不幸成了“凶器”。我们办公室里面，另外比较显眼的东西就是墙上各种各样的照片，除了生日墙那些“入狱照”，剩下的基本都是大家偷拍或者搞怪的照片。

在ThoughtWorks，有一条不成文的规定，迟到的人要给项目组的其他人买冰淇淋。运气好的话，连续几天都有冰淇淋吃，在大夏天，这可是一件让人高兴的事。当然，另一方面，这也说明了，有人破坏了我们的纪律。冰淇淋纪律既让大家知道纪律的存在，也为我们的开发平添了几分乐趣。

我们的办公室里面，经常会有笑声，不用问，肯定是有人出了些搞笑的段子。有人无意间雷倒众生，也有人为大家制造着笑料。一些内部推手通过各种手段，很快把这些笑料爆出去，于是，这就成了全办公室尽人皆知的“秘密”。

有时侯，身为开发人员，我们偶尔会欺负一下“项目组中地位最低”的PM，他就要在工作时间跑出去为我们买一些零食，比如最近一段时间非常受欢迎的豆腐皮和花生米，以保证我们安安心心的写代码。赶上回顾会议，我们就更有理由一边吃着零食，一边“回顾”了。在我出去做咨询的时侯，甚至把这种风格带到了客户那边。

Team Building在项目组里面永远是特别受欢迎的一项活动。吃饭、唱歌是最常见的活动，当然发掘新鲜的活动项目是一个永远没有止境的话题。最近一段时间经济不那么景气，不再像之前那样，公司能够为项目组提供的预算大幅度缩减了，但即便是大家AA，我们依然可以自得其乐。

软件开发本身已经有了一定困难，与其每天苦着脸，还不如“苦中作乐”，至少我们可以有一个良好的心情去面对那些困难的问题。

“高高兴兴上班，平平安安下班”，这是小时侯经常在一些工厂看到的宣传标语，人们的愿望其实很朴素。

上海肯德基20年Say Yes-光阴的故事

Manish Chakravarty: Confusion Confusion

Sun, 05 Jul 2009 05:40:40 +0000

1. TW me paisa kam par nice profile and I love the job

2. Nokia me paisa zyaada, international exposure but QA profile.

Which one should I choose?

I am confused about my career for the first time in life.. any inputs would be good.

Jeff Xiong: AgileChina 2009: Pragmatic Agile

Sun, 05 Jul 2009 02:37:20 +0000

AgileChina 2009大会官方网站

由在敏捷领域最具有影响力的技术社区InfoQ中文站、敏捷方法论的领导厂商 ThoughtWorks共同主办的敏捷中国技术大会（Agile China 2009），将于9月11日~12日（周五、周六）在北京举行。届时将有超过500人来自电信、金融、互联网、教育等行业在内的高级软件开发人员、项目管理人员等参加。本次大会将特别邀请敏捷宣言缔造者、敏捷编程（XP）方法学创始人Kent Beck，敏捷开发权威人士、敏捷宣言的创始人之一，Dave Thomas，敏捷宣言签署人之一Steve Freeman等国际敏捷领域专家，以及在团队中成功应用敏捷的阿尔卡特、赛门铁克、诺基亚-西门子、华为、腾讯等公司的项目负责人参与此次大会并分享他们的心得。
敏捷中国技术大会（Agile China 2009）是国内敏捷技术领域最高水平的大会，本次大会将由InfoQ中文站负责大会策划、营销和项目实施。InfoQ中文站在今年4月已成功举办了 QCon全球企业开发大会，邀请了国内外30多名讲师，超过550位架构师、技术总监、项目经理和高级工程师参加了本次大会，大会及其运营团队获得了空前的好评，并誉为国内“技术含量最高”的大会。而今年的敏捷中国技术大会，也将一改往年的风格，参会者以高端开发者和技术管理者为主，融合管理和工程实践，推广全面敏捷之路。

会务咨询： 010 – 89880682 agilechina@cn.infoq.com

赞助咨询： 010 – 89880682 sponsor@cn.infoq.com

Danilo Sato: Velocity gone wrong #2: Making up points

Sat, 04 Jul 2009 20:18:14 +0000

Continuing with the series on how to misuse velocity, the second anti-pattern I would like to highlight is when teams start making up points. Because the definition of velocity is so simple, it’s easy to game the metric to show what looks like apparent progress. If a team is being measured on velocity (more about this on later posts), it’s quite easy to just start increasing estimates: “If we just double all estimates, the relative sizes stay the same, but our velocity doubles!“. This is an extreme behaviour that would be quickly noticed as a discrepancy, but the same thing could happen in a smaller scale and pass unnoticed.

This problem can not only be originated from the team, but I’ve also seen Project Managers/Scrum Masters coming up with “clever” ways of making up points to count as velocity:

Counting percentage or half points (as mentioned in my previous post)
Deciding to split a story to count the partially finished work as complete, and track whatever is left in a separate story (splitting should be business-driven and not tracking-driven: it should only happen when you come up with simpler/incremental ways of delivering value in smaller chunks)
Counting points on technical tasks. I’ve seen a team that spent a lot of effort in an iteration to make up for accumulated technical debt, and did not have a lot of time to work on new stories. The Project Manager decided to come up with a “refactoring card” and gave it a 16 to try and demonstrate how much effort was spent on such refactoring
Counting points for in-release bug fixing. In a team, stories were deemed completed on the first iteration, but bugs started to show up in later iterations, impacting he team’s ability to deliver new functionality. Instead of allowing the decrease in velocity to demonstrate how the lack of focus on quality was impacting the team (bugs should be prevented in the first place, right?), the Project Manager decided to estimate and count points on bugs, which kept velocity apparently constant, when in fact a lot less value was being delivered

The next time you catch yourself asking “Should X count as velocity?”, stop, reflect, and ask instead “Should I worry about X happening at all?”. If you are worried about having to track or show progress on things that should be embedded parts of the process (such as activities to prevent bugs or refactoring), chances are that the problem lies somewhere else. Some of these questions might make as much sense as “Should time spent on retrospectives count as velocity?” or “Should going to the bathroom count as velocity?” :-)

I’m sure that these examples drawn from my personal experience are just a few examples of how to make up points and misuse velocity. What other similar experiences did you have in your own projects?

Jason Yip: 3 imperfect strategies for system test data management

noreply@blogger.com (Jason Yip) — Sat, 04 Jul 2009 19:08:15 +0000

Automatic discovery of input data
Configurable input/output expectations and controlled data refreshes
Automated scripts to manipulate/insert data after each refresh

Jason Yip: Some thoughts on roles and hand-offs

noreply@blogger.com (Jason Yip) — Sat, 04 Jul 2009 19:01:25 +0000

Here's a typical role and hand-off sequence that I encounter in "waterfall" shops:

Customer -> Business Analyst -> Developer -> Tester

And then they learn about Agile so the sequence changes to this:

Customer -> Business Analyst -> Tester -> Developer -> (Business Analyst + Tester)

... which really misses the point. We're not just re-ordering hand-offs, we actually want to remove them.

What I'm looking for is a sequence that looks more like this:

(Customer + Business Analyst + Tester + Developer) -> (Tester + Developer (referencing Business Analyst and Customer as useful)) -> (Customer + Business Analyst + Tester)

If the Customer time is constrained,

Customer -> (Business Analyst + Tester + Developer) -> (Tester + Developer) -> (Business Analyst + Tester) -> Customer

Sam Newman: Camera has arrived. eBay, thou art the wind beneath my wings.

Sat, 04 Jul 2009 14:55:00 +0000

Camera has arrived. eBay, thou art the wind beneath my wings.

Simon Brunning: Links for 2009-07-03 [del.icio.us]

Sat, 04 Jul 2009 07:00:00 +0000

Geisha

Mark Needham: Domain Driven Design: Conformist

Sat, 04 Jul 2009 00:17:31 +0000

Something which constantly surprises me about Domain Driven Design is how there is a pattern described in the book for just about every possible situation you find yourself in when coding on projects.

A lot of these patterns appear in the 'Strategic Design' section of the book and one which is very relevant for the project I'm currently working on is the 'Conformist' pattern which is described like so:

When two development teams have an upstream/downstream relationship in which the upstream has no motivation to provide for the downstream team's needs, the downstream team is helpless. Altruism may motivate upstream developers to make promises, but they are unlikely to be fulfilled. Belief in those good intentions leads the downstream team to make plans based on features that will never be available. The downstream project will be delayed until the team ultimately learns to live with what it is given. An interface tailored to the needs of the downstream team is not in the cards.

We are working on the front end of an application which interacts with some services to get and save the data from the website.

We realised that we had a situation similar to this originally but didn't know that it was the conformist pattern and our original approach was to rely completely on the model in the service layer to the extent that we were mapping directly from SOAP calls to WCF message objects and then passing these around the code - I originally described this as being an externally defined domain model.

This led to quite a lot of pain as whenever there was a change in the service layer model our code base broke all over the place and we then ended up spending most of the day fire fighting - we were too tightly coupled to an external system.

At this stage we were reading Domain Driven Design in our Technical Book Club and I was fairly convinced that what we really needed to do was have our own model and create an anti corruption layer to translate between the service layer model and the new model that we would create.

We changed our code to follow this approach and created repositories and mappers which were the main places in our code base where we cared about this external dependency and although the isolation of the end point has worked much better we never really ended up with a rich domain model that really represented the business domain.

We had something in between the service layer model and the real business model which didn't really help anyone and meant we ended up spending a lot of time trying to translate between the different definitions that were floating around.

Writing the code for the anti corruption layer also takes a lot of time, is quite frustrating/tedious and it was hard to see the value we were getting from doing so.

We've now reached the stage where we know this is the case and that it probably makes much more sense to just accept it and to not spend any more time trying to create our own model but instead just adapt what we have to more closely match the model we get from the services layer.

We will still keep a thin mapping layer as this gives us some protection against changes that may happen in the service layer.

I think a key thing for me here is that it's really easy to be in denial about what is actually happening since what you really want is to be in control of your own domain model and design it so that it closely matches the business so that they would be able to read and understand your code if they wanted to. Sometimes that isn't the case.

Chatting with Dave about this he suggested that a lesson for us here is that it's important to know which pattern you are following which Andy Palmer also pointed out on twitter.

Mark Needham: Coding Dojo #19: Groovy Traveling salesman variation

Fri, 03 Jul 2009 23:36:01 +0000

Our latest coding dojo involved working on a variation of the traveling salesman problem in Groovy again.

The Format

We had 8 people participating this week so we returned to the Randori format, rotating the pair at the keyboard every 7 minutes.

Give the number of people it might have actually been better to have a couple of machines and use the UberDojo format.

What We Learnt

The importance of just getting started stood out a lot for me in this dojo - there have been quite a few times when we've met intending to do some coding and spent so long talking about coding that we didn't end up writing anything. Luckily Dave took the lead in this dojo and got the ball rolling. The code we wrote originally wasn't perfect but it helped create the momentum to keep the session going so it was valuable in that way.
Another interesting feature of dojos for me is that it really doesn't matter if you make mistakes - if you write really terrible code in a dojo it's probably a good thing since you'll probably not go and repeat the same mistake on a real project. I learnt a lot about the perils of not refactoring early enough and having too much state in our code from our Isola Dojo a few months ago.
We refactored much earlier than we normally do in this dojo and I think it worked really well for allowing us to progress later on. Often we fall into the trap of just chasing the green bar a bit too much and we forget to clean up the code after each cycle but we had that a bit better in this one.
We also backed up a bit after around 3 cycles after realising that the code was becoming a bit horrific and spent 1 cycle working it into shape for the next one.
We fell into the trap of going several cycles with broken tests while trying to do some redesign on the code - the steps were clearly not small enough!
Later on we corrected this when refactoring the code into a more functional style by taking very small steps and running the tests after each small change - this was a far more effective approach.
Although we were working in a dynamic language it didn't feel that the conversations were that different when discussing the code - we were still talking about types when working out what to do. I'm not sure whether this means we haven't quite got the idea of dynamic languages or whether there isn't such a big difference between the way you talk about your code in them.

For next time

We might continue with another problem in Groovy - it's been quite fun working in a language that runs on the JVM without the verbosity you sometimes get when writing Java code.

Jason Yip: Don't just make it visible, make it tangible

noreply@blogger.com (Jason Yip) — Fri, 03 Jul 2009 19:58:04 +0000

I was talking to Erik yesterday and he was wary that I used the phrase "make the internal quality issues tangible" and said he preferred visible or explicit.

I suggested that explicit and visible aren't actually enough. I truly mean "tangible" because the particular situation I'm targeting is the non-technical audience that doesn't understand what is happening with the long-term health of their systems due to short-term project trade-offs.

Explicit is a number which is too abstract. Visible is better because then they can see. But I actually want people to feel, emotionally, the danger which they are putting themselves in.

Our target is tangible with visible being along the way there.

Patrick Kua: Be Careful What You Measure

Fri, 03 Jul 2009 10:59:55 +0000

Danilo writes about, “Done is not done” and I’m reminded of being careful about using the right metric. I remember being on a project where people had individual velocity. This is a really bad thing unless you want to prevent collaboration. I’d add it to Danilo’s list.

Anand Vishwanath: Challenges in applying Theory Of Constraints to Agile Software Development

noreply@blogger.com (Anand Vishwanath) — Fri, 03 Jul 2009 09:30:11 +0000

After finishing Goldratt's first book, the Goal, I immediately jumped to David Anderson's book on Applying TOC in Agile Management. Preetam also has left some interesting comments on my blog on the differences between s/w and manufacturing which should not be neglected.

One important thing which is still puzzling me at this stage is, if you have to account for throughput, how do you measure value. Value here is the value of a story or a feature. If you are a services shop building an application for a customer, it is upto the customer to determine this business value of a story. Quite often this is not easy to determine, specifically for people like Subject Matter Experts you are dealing with to gather your requirements, who do not deal with sales at all.

In a manufacturing plant, if you have signed a contract to deliver 10 cars for say $100,000. The cost of an engine still waiting in the queue (work in progress) is almost equal to the value of 1 car, $10,000. However the similar analogy is not easily applicable to a software development team. If a story is in progress, it cannot be easily translated to how much $$ is part of inventory.The story might be part of a bigger feature. Also, determining the business value for that feature in $$ itself can be tricky

David's book touches upon in-house IT shops providing IT services (eg: SaaS) or implementing a software product. In these cases, revenue from Sales can be directly mapped to what is being developed. Which is what should happen, in order to increase the throughput of the system, and hence increase the revenue. This is not the case when you are providing IT services to other clients (not in-house).

Some might say you can use story estimate (ideal days or points). Again this represents cost and not value. The iterative nature of Agile also makes it difficult to apply the same principles as a manufacturing plant.

While I am still looking for some answers, the learning on how to deal with bottlenecks, optimizing the whole system, and applying the same to software development, from both the books, are still quite useful.

Danilo Sato: Velocity gone wrong #1: Done is not done

Fri, 03 Jul 2009 00:12:15 +0000

Dan North wrote an interesting post about the perils of estimation, questioning our approach to inceptions, release planning, and setting expectations about scope. This made me think about the implications of those factors once a project starts, and I came up with some anti-patterns on the usage of velocity to track progress. This is my first attempt at writing about them.

Before we start, it’s important to understand what velocity means. My simple definition of velocity is the total number of estimation units for the items delivered in an iteration. Estimation unit can be whatever the team chooses: ideal days, hours, pomodoros, or story points. The nature of items may vary as well: features, use cases, and user stories are common choices. Iteration is a fixed amount of time where the team will work on delivering those items. Sounds simple? Well… there’s one concept that is commonly overlooked and that’s the source of the first anti-pattern: what does delivered means?

One of the most common anti-patterns I’ve seen is not having a clear definition of done. Lean thinking tells us that nothing is really done until it’s delivering value, which in software means: code running in production and being used by real users. Although I know very few teams who can deploy code to production at the end of every iteration (some even do more than once per iteration), once a story is considered done, it could be potentially shipped, if the business decides so. There shouldn’t be a lot of extra work after that.

Another bad implication of this anti-pattern is that some teams decide to change the definition of done and count half-completed work to show progress. Some of the symptoms to help diagnose if your team is suffering from this anti-pattern are:

The team starts tracking dev-complete stories
“It’s done, but [we need to write the acceptance test/it's not integrated with the other system/...]“
“It’s done, but not done-done”
It takes a lot of extra work to get the story deployed to production
After finished, the story goes into the next team’s backlog
Hearing terms like “development team velocity” or “test team velocity”
Counting half-points or percentages because “if we don’t count it will look like we haven’t worked”

The solution? Remember that velocity is just a number that provides information for the team to understand and improve it’s process. Forget that you’re tracking it and focus on the entire Value Stream and on what’s really value-added to get things into production. Anything else is just waste. If it’s not done, it’s not done. Accept it, move on, and don’t overcomplicate, because it will only add noise and mask what could have been important information to the team.

Nick Drew: nickdrew

Thu, 02 Jul 2009 23:26:41 +0000

I’m going to tell you that, with a bit of effort, you can get a really good Test Driven Red/Green cycle with Silverlight…

Assumptions – you have installed VS 2008, and the Silverlight 2 SDK

Firstly – we created 4 projects in Visual Studio:

Silverlight.App – holds the production code
Silverlight.App.UnitTests – runs tests that require no external dependencies
Silverlight.App.IntegrationTests – contains tests that connection to the web, database, are asynchronous, etc
Silverlight.App.IntegrationHost – a ASP.NET project that will run the IntegrationTests

UnitTests and IntegrationTests are created with the Silveright UnitTesting project templates. These templates cause a test page to be created. The test page basically listens to events from the TestRunner (embedded in the Test silverlight applications), and presents it as (fiddly) HTML.

Our UnitTest project hasn’t changed much since we started. We have had issues with the IntegrationTests project:

Security

Code transparency – All application logic is marked as “Transparent”, which means that may call but may not extend or override “SafeCritical” or “CriticalCode”. The System.Net.WebClient is marked as SafeCritical. This means that you won’t be able to Mock/Stub WebClient. At all.
Internal event constructors – The constructor for DownloadStringAsyncCompleteEventArgs is internal; You won’t be able to simulate a WebClient completing an http request.
Silverlight applications loaded from the file system just plain cannot connect to web resources. There is no security policy that will allow it.

All of the above issues drove us to use the IntegrationHost to run the IntegrationTests, but in a web context. This is quite good, especially for within-IDE testing, since you can host your test data as flat files or as ASP driven test data in the same host.

Unit Testing

Good news

Rhino mocks is available for Silverlight.
Nunit is available as source for Silverlight (with some shims for older, pre-silverlight data structures).
The Silverlight UT runner works with both MSTest Metadata and nunit style assertions at the same time

Puzzling

Nunit lite appears to be at version 0.5, and maybe worth looking at instead of a custom build of Nunit
Specificity Looks like it may work in the Silverlight environment.
Asynchronous testing using the [Asynchronous] attribute is potentially very brittle
Maintaining the test data for the integration tests in the IntegrationHost web project seems counter intuitive

Continous Integration

Silverlight UT test reports are shockingly bad HTML, making downstream consumption particularly difficult. Would be nice to log semantic html somewhere.
For the unit tests, we found a powershell script that will spawn IE and scrape the results into a file
For the integration tests, we modified the powershell script to spawn a WebDev.WebServer.exe against the IntegrationHost
The IntegrationTest xap file that is linked from the IntegrationHost will not get automatically deployed by msbuild – we use a copy task to pull it before launching IE.
Visual Studio wants to put the hosted xap file in ClientBin of the IntegrationHost. This is fine, but it also wants to put it into source control, which is not. Eventually, you’ll get the right combination of check-ins, deletes, et. al. and it won’t be an issue. Expect to lose a few hours on this.

Powershell

When you have to clear out some space for that web server socket:

gwmi win32_process | where { $_.CommandLine -like "*WebDev.WebServer.exe*/port:$port" } | Kill

Asynchrony

Async silverlight tests look a bit like this:

[Asynchronous]
public void ShouldHangAroundABitWaiting {
   EnqueueCondition( () => return TrueIfCanStart() );
   EnqueueCallback( () => Specify.That( something, Is.Ready() );
   EnqueueTestComplete();
}

It is at times like this that one definitely wants a monad.

One day, async code will look like this:

[Asynchronous]
public void ShouldHangAroundABit {
	TrueIfCanStart.Wait();
	Specify.That( something, IsReady() )
}

Steven List: IAAM: Sympathizing, Empathizing, Identifying

Thu, 02 Jul 2009 19:01:38 +0000

[This is fiction. Any resemblance to individuals living or otherwise is purely coincidental. Really.]

Joan’s phone starting ringing insistently. Joan thought for a moment, since she was watching her favorite reality TV show, and that was her time to just disconnect. In spite of her preferences, Joan decided to answer the phone.

“Joan?” She heard her friend Nancy’s voice, and her heart skipped a beat. Nancy was sobbing. “Nancy? What’s wrong honey?”

“They fired me, Joan! They fired me!”

“Joanie…” sobbing “…they said that I just wasn’t living up to their expectations.”

“Oh, Nancy…”

Now let’s talk about Joan’s possible reactions…

Each of us has a different reaction, and each of us offers a different response based on that reaction*. For the moment, I want to talk about three types of reaction and response: sympathy, empathy, and identification.

Reaction: Sympathy

Response: “Oh, Nancy… that’s terrible. You must feel miserable. I can only imagine how that feels. Would you like to come over and talk?”

Reaction: Empathy

Response: “Oh, Nancy… I feel terrible. I can’t believe it! I’ll come over and let’s talk about what we can do.” Joan cries.

Reaction: Identification

Response: “Oh, Nancy. Those bastards! After all you’ve done for them, and how hard you’ve worked. You gave your all to that company, and this is how they treat you? I’m devastated.”

Nancy cries.

Joan cries.

Up to this point, sympathy, empathy, and identification sound a lot alike. In all three versions, Joan has an emotional reaction that leads to a behavior – her response. In each case, her response is subtly different. Note that in the following discussion, I am not making a judgment about better versus worse, or good versus bad… I’m working on achieving understanding and recognizing that each type of reaction and response deserves and requires a different response from me.

In being sympathetic, Joan’s response is separate. Joan is clear that what is happening to Nancy is about Nancy, not about Joan. While Joan may feel sad or angry, it is on behalf of her friend. From Nancy’s perspective, there is a little bit of distance between them. Joan’s feelings are moderate.

In being empathetic, Joan’s response is collective. Joan feels what she believes Joan feels, including the pain, indignation, and so forth. For Joan, what is happening is also happening to her, emotionally. From Nancy’s perspective, it’s like a resonation, which may increase the level of her feelings. To a certain extent, Joan’s reaction becomes an extension of Nancy’s reaction. Joan’s feelings are intense, although she recognizes that they are about Nancy.

In identifying with Nancy, Joan takes on Nancy’s feelings and reactions. Joan’s response is intense and personal, as though she were the one who had been fired. Nancy may be taken aback by the intensity of Joan’s reaction, as Joan takes on some of Nancy’s emotional response. Joan behaves as if she were the one who had been fired, and will react to others as if she were the victim as much as Nancy.

To see how this works, let’s add Joan’s husband Mark to the story…

“Joan? What’s going on?”

Sympathy

“It’s Nancy. She got fired today. I feel so bad for her. She’s so upset.”

“That sucks. What’s she going to do?”

“I don’t know yet. I may have to spend some time with her. I hope that’s okay with you.”

Empathy

“It’s Nancy. She got fired today. I’ve got to go over there to be with her right now!” Sobbing

“That sucks. What’s she going to do?”

“I don’t know yet, but I just know how horrible she feels and that I have to go be with her. It’s so painful! Doesn’t this upset you?”

Identification

“It’s Nancy. She got fired today. I’ve got to go over there to be with her right now!” Sobbing

“That sucks. What’s she going to do?”

“They treated her like dirt! How can you be so calm? Don’t you care? They were unfair and cruel. I don’t know what we’re going to do, but we’re going to do something to show them!”

Note that Joan’s response to Mark escalates from Sympathy to Empathy to Identification. In the latter, Joan feels that what has happened to Nancy has happened to her, and thus she expects the same kind of reaction from Mark that she’d expect if she had been fired.

This post is long enough. Now I’m going to go off and think about the differences in responses to each of the three.

Sympathy

an inclination to support or be loyal to or to agree with an opinion; “his sympathies were always with the underdog”; “I knew I could count on his …
sharing the feelings of others (especially feelings of sorrow or anguish)
a relation of affinity or harmony between people; whatever affects one correspondingly affects the other; “the two of them were in close sympathy”

wordnet.princeton.edu/perl/webwn
http://dictionary.reference.com/browse/sympathy

Empathy

understanding and entering into another’s feelings

wordnet.princeton.edu/perl/webwn
http://dictionary.reference.com/browse/empathy

Identification

the attribution to yourself (consciously or unconsciously) of the characteristics of another person (or group of persons)

wordnet.princeton.edu/perl/webwn
a process by which one ascribes to oneself the qualities or characteristics of another person.
A person’s association with or assumption of the qualities, characteristics, or views of another person or group.

http://dictionary.reference.com/browse/identification

*Reaction vs. Response

For the purposes of this discussion, I’m defining “reaction” as the emotional or physical effect that occurs without thinking, and “response” as the chosen action or thought that occurs after the reaction. That is, if I put my hand in a fire, pulling my hand out is a reaction – I don’t think about it – while swearing about it is a response.

Tweet This Post

Sam Newman: I like scala, but binary incompatibility between minor versions? Come on! http://bit.ly/ScalaXBuild

Thu, 02 Jul 2009 18:41:27 +0000

I like scala, but binary incompatibility between minor versions? Come on! http://bit.ly/ScalaXBuild

Andy Palmer: Pomodoro, AppleScript and Adium

Thu, 02 Jul 2009 15:21:56 +0000

I downloaded this Pomodoro Timer from the Apple website.

It has support for AppleScript events, so I created a script that automatically sets my Adium status to away (with an auto-reply) while I am working on a Pomodoro and automatically sets it to Available when I have finished.

After much messing around, trying to work out just how the AppleScript pseudo-natural-language works, I ended up with this to set the away status:

-- This goes on one line in Pomodoro Setup -> AppleScript -> Start
tell application "Adium"
  to set the status of every account whose status type is available
  to the first status whose title is "Pomodoro In Progress"

and this to set it back to available:

-- This goes on one line in Pomodoro Setup -> AppleScript -> Reset and End
tell application "Adium"
  to set the status of every account whose status type is away
  to the first status whose title is "Available"

Mark Needham: F#: Pattern matching with the ':?' operator

Thu, 02 Jul 2009 13:10:19 +0000

I've been doing a bit more reading of the Fake source code and one interesting thing which I came across which I hadn't seen was an active pattern which was making use of the ':?' operator to match the input type against .NET types.

  let (|File|Directory|) (fileSysInfo : FileSystemInfo) =
    match fileSysInfo with
      | :? FileInfo as file -> File (file.Name)
      | :? DirectoryInfo as dir -> Directory (dir.Name, seq { for x in dir.GetFileSystemInfos() -> x })
      | _ -> failwith "No file or directory given."

I thought maybe this was just a wild card operator to say that we don't care what the value is as long as it matches 'FileInfo' or 'DirectoryInfo' respectively but I couldn't see it defined on the list of operators on the Microsoft Research website.

A bit of googling led me to Matthew Podwysocki's post about pattern matching which explained the purpose of the operator (about 1/3 of the way down):

What the above example does is check for the corresponding .NET types by using the ':?' operator especially reserved for this behavior.

I've been playing around with a simple 'add' function to try and understand F#'s type inference and one thing I noticed is that if you just define it with minimal code you end up with a function which takes in 2 integers and returns an integer as the result:

let add a b = a + b
 
val add: int -> int -> int

I had thought that the signature and result of that function might remain generic due to the fact that there are more types than just 'int' with which you can make use of the addition operator.

For example, it is possible to add two string together but in fact you need to be more explicit about that:

let add (a:string) (b:string) = a + b
 
val add: string -> string -> string

From what I can tell if we wanted to write a generic add function we would need to do something like this - I originally tried just returning 'new A + new B' from each of the pattern matches but the return type of add3 then becomes 'string' since the first path in the pattern matching returns a 'string'.

    let add3 a b =
        match (box a,box b) with
            | (:? string as newA),(:? string as newB) -> newA +  newB |> box
            | (:? int as newA),(:? int as newB) -> newA + newB |> box
            | (:? decimal as newA),(:? decimal as newB) -> newA + newB |> box
            | _ -> failwith "you can't add these together"

Which is slightly verbose and has a type of "'a -> 'b -" obj' - I haven't been able to work out whether it's possible to create a generic function like this without needing to cast the result down to 'obj'.

I thought it might be possible to get rid of the boxing by making use of the downcast operator:

You can also use the downcast operator to perform a dynamic type conversion. The following expression specifies a conversion down the hierarchy to a type that is inferred from program context.

I tried surrounding the 'newA + new B |> box' code with a call to 'downcast' but that just resulted in the following error message when trying to make use of the function:

Value restriction. The value 'it' has been inferred to have generic type
	val it : '_a
Either define 'it' as a simple data term, make it a function with explicit arguments or, if you do not intend for it to be generic, add a type annotation.

I'd be intrigued to see if anyone has worked out how to do this as I'm out of ideas.

Andy Palmer: Tiny Types

Thu, 02 Jul 2009 12:29:32 +0000

Mark Needham’s post on micro types sparked an idea in my head that recently came to fruition.

I was pondering on the Narrative Fixture code, and the fact that, although most of the internals are sensibly typed objects, at the FitNesse layer, we do a lot of passing of strings.

The idea (at least, my interpretation) with Tiny Types is that the string obviously means something, and that we can probably classify what that meaning is (we have a sensibly named variable for it, after all) , and so why not convert it to that meaningful thing as soon as possible.

This has the nice effect of making some vague things (such as typing on collections) much more explicit.

As an example, in the Casting Director we have:

Map<String, ActorPlayingThe<?>> dressingRoom = new HashMap<String, ActorPlayingThe<?>>();

and, after a little bit of refactoring, we end up with:

Map<CharacterName, ActorPlayingThe<?>> dressingRoom = new HashMap<CharacterName, ActorPlayingThe<?>>();

Personally, I’m a fan of low ceremony, high essence languages, but while working in a high ceremony environment, we can leverage that ceremony to provide us with a nice summary of our thinking, ready for the next time we return to the code.

Sam Newman: Is #mongodb too good to be true? Where’s the catch? Ah - their website is powered by...

Thu, 02 Jul 2009 08:27:01 +0000

Is #mongodb too good to be true? Where’s the catch? Ah - their website is powered by Confluence…

Mark Needham: Book Club: Logging - Release It (Michael Nygaard)

Thu, 02 Jul 2009 02:04:34 +0000

Our latest technical book club session was a discussion of the logging section in Michael Nygard's Release It.

I recently listened to an interview with Michael Nygard on Software Engineering Radio so I was interested in reading more of his stuff and Cam suggested that the logging chapter would be an interesting one to look at as it's often something which we don't spend a lot of time thinking about on software development teams.

These are some of my thoughts and our discussion of the chapter:

An idea which Nick introduced on a project I worked on last year was the idea of having a 'SupportTeam' class that could be used to do any logging of information that would be useful to the operations/support team that looked after our application once it was in production.
This is an approach also suggested by Steve Freeman/Nat Pryce in Growing Object Oriented software (in the 'Logging is a feature' section) and the idea is that we will then focus more on logging the type of information that is actually useful to them rather than just logging what we think is needed.

One thing which Dave pointed out is that it's often difficult to get access to the operations team to try and get their requirements for the type of logging and monitoring they need and so often ends up being something that's done very late on. On projects I've worked on there has often been a story card for logging and I think this is a good way to go as they are a stakeholder of the system so logging shouldn't just be dealt with as a nice extra.
Something which I hadn't considered until reading this book is the idea of making logs human readable and machine parseable as well. The default format of most of the logging tools is not actually that useful when you're trying to scan through hundreds of lines of data and it was intriguing how a little indentation could improve this so dramatically with the added benefit of making it much easier to create a regular expression to find what you want.
One thing I'm interested in understanding is how we work out what's too much logging and what's too little since it seems that it seems that the answer to this question is fairly context sensitive. For example on a recent project we logged all unhandled exceptions that came from the system as well as any exceptions that happened when retrieving data from the service layer. In general the data we've had available has been enough to solve problems but we could probably have done more, just working out what would be useful doesn't seem obvious.
I think it was Alex who pointed out that it's often useful to have an explicit step in the build to remove any debug logging from the code so that it doesn't end up in production by mistake. This seems like a pretty neat idea although I haven't seen it done yet - it also leads towards the idea that logging is for the operations team which I think is correct although it is often suggested that logging is actually for developers since it is assumed that they would be the ones to eventually solve any problems that arise.
The idea of having message codes for specific errors messages seems like a really cool idea for allowing easy searching of log files - we've done this on some projects I've worked on and not on others. I guess the key here is to ensure we don't end up with too many different error codes otherwise it's just as confusing as not having them at all.

Dan North: The perils of estimation

Wed, 01 Jul 2009 22:50:38 +0000

Business people want estimates. They want to know how much it’s going to cost them to get a solution, and they want to know how likely it is to come in on time and on budget. And of course quality is not negotiable.

Agile teams I encounter are at best nervous about estimates and at worst simply evasive. “You don’t need estimates if you’re doing Agile,” they say. “It will be ready when it’s done. We’re constantly adding value so we don’t need to commit to a date.”

We’re missing the point of release planning

My favourite exchange goes something like:

“We’ve done an inception and broken down the entire project into stories and measured it, and it’s come in at 400 stories, estimated at 865 story points.”

“865 what?”

“865 story points.”

“So how big is a story point?”

“We don’t know yet, we’ll let you know in a few weeks.”

At a governance and funding level the business could care less about story points. They don’t actually care about stories except that we shove them in their faces with our release plans. They care about solving a problem. They came to us and asked us a) how much will it cost to solve the problem, and b) how confident are we about that number?

So how do we approach that? We go through some sort of inception process that looks something like this:

1. Identify some personas

2. Identify some process flows

3. Start breaking the flows down into stories

4. Lots and lots of stories

5. Lots and lots and lots and lots of stories

6. Spike some technical ideas that came out of the stories

7. Estimate the stories

8. Roll up all the estimates and call that our project estimate

The part where we estimate the stories is a real chore (c’mon – we’re estimating 400 stories here), so we cut corners. We do a first pass as t-shirt sizes (small, medium, large) and then take a representative sample (sounds suitably scientific) and do a “detailed” estimate of those. This involves a bunch of people estimating lots of important-sounding metrics: minimum, likely and maximum size, clarity, volatility (eh?) and whatever else, and then multiply it up to provide a WOOOOAAAAAHHHH! hang on a minute! What were we trying to do again?

All they wanted to know is: How long will it take, and how confident are you about that number?

Redefining success – in a bad way

By introducing a comprehensive story list – with or without the notion of story points – we have unwittingly reframed the project. The business started out by defining success as solving the problem, but now we have redefined success as delivering this list of stories. However we frame it, that’s what the business will believe. The project will start and and the business stakeholders will start counting down the list of stories until you get to zero.

So now we have the worst of both the Agile and plan-driven worlds: the business expects delivery of a fine-grained list of requirements (whether we call it a Product Backlog or a Master Story List), and we have only taken a half-hearted attempt at it compared to the big up-front analysis we used to do. From here on we are on the back foot, constantly negotiating with the business to manage scope, when it’s our own fault they even care about the story-level detail. They see the story backlog and mentally turn it 90 degrees and think of it as a Gantt chart. Happy days!

Back to project basics

There are two observations to make here. Firstly the business wants accuracy and we’re giving them precision. If you tell me it will take 4.632 months and it takes 8 months, that’s worse than useless. If you tell me it takes “about six months” and it takes seven months, I should still be onto a winner. (If the return on investment is small enough that the extra month stops the proposition being viable, I’d have been better off investing in something else in the first place. Spending $60,000 to realise a return of $70,000 is risky to say the least.) I’m simplifying here of course because the real RoI varies over time, and its value may be particularly time-sensitive.

Secondly we know that requirements vary on an agile project over time, for good reason. Hopefully we are learning as we go, which means we will discover new requirements and decide others are no longer worth pursuing. If we assume about a third of the requirements will be delivered as described (this is generous in light of the Standish Chaos reports), another third will be delivered but with changes, and the last third won’t be delivered at all – but replaced by other features – then we have just wasted all that time and effort in the inception coming up with detailed, high-_precision_ data for a pile of stories we will never deliver.

To compound this, it turns out that estimation is fractal. The more fine-grained you break down the requirements, the more “edges” you will discover. This means that the more detailed you estimate, the more the total will tend towards infinity, simply due to rounding errors and the fear factors that we multiply into fine grained estimates.

Use the inception for deliberate discovery

So what should we be doing during an inception instead of doing the fine-grained story breakdown? Taking it back to first principles we simply want a rough idea of size and an understanding of certainty. There is uncertainty in everything, so the purpose of the inception is to understand the potential landscape we are delivering in.

When we start the inception we know nothing. We want to come out of the inception knowing as much as we could reasonably expect to learn in the time we have allocated. This discovery is along several axes: technical areas such as the technology stack, potential architectures, integration points and external services; domain questions such as how well we understand the problem and whether it is more about research than solving a clearly-articulated problem; people and process challenges like the path to production, identification of stakeholders, how co-located or distributed the team is and how much of this kind of delivery they have done before. For some of these areas, breaking broad requirements into finer-grained detail is a great way to discover more. But not for all of them, and certainly not at the expense of other discovery activities.

There are good arguments from both the Kanban and Real Options folks about deferring the decomposition of feature sets into features and stories until the “last responsible moment”. This means the information is freshest and you aren’t holding an inventory of atrophying information. You might want a couple of weeks of story-level detail – to promote a consistent flow and avoid starving your process – and beyond that a few features identified that will be broken down into the next candidate stories, but beyond that you shouldn’t be worrying about that level of granularity. The experienced members of the team should be estimating feature sets of the order of person-weeks (or better yet, pair-weeks), not going down to the level of individual pair-days. The less experienced team members should be using the exercise as a learning opportunity.

So please, let’s move beyond this cargo cult approach to inception where we slavishly trot out hundreds of stories with their associated estimates, and remember that we are engaging in a process of deliberate discovery.

Its purpose is firstly to convey to our stakeholders and ourselves an order-of-magnitude sense of size – to quote the Pragmatic Programmers, is it larger than a breadbox and smaller than a house? – and secondly to present the risk landscape in which to understand that estimate.

Pankaj Nakhat: My Childhood

Wed, 01 Jul 2009 21:25:15 +0000

Sam Newman: Just watched: #Yellowstone on BBC4. Simply stunning - rivals Planet Earth for beauty. No idea why...

Wed, 01 Jul 2009 21:12:53 +0000

Just watched: #Yellowstone on BBC4. Simply stunning - rivals Planet Earth for beauty. No idea why it’s hidden on BBC4…

Martin Fowler: RequestStreamMap

Wed, 01 Jul 2009 18:58:00 +0000

Hang around my colleagues at ThoughtWorks and you soon get the impression that the only good Enterprise Service Bus (ESB) is a dead ESB. Jim Webber refers to them as Egregious Spaghetti Boxes. So it's not uncommon to hear tales of attempts to get them out of systems that don't need them.

Battle was joined at one client and it brought to mind my younger days playing D&D. Webber swings but misses as the ESB is AC 2, Evan gets a hit and rolls 2d8 for 6 damage. Erik finally kills it by casting "Summon Request Stream Map".

So what was Erik's decisive spell? Essentially the idea was to take a simple request and show how the data for the request and response made their way through the layers of the application. Erik printed out all the code that you needed to read to understand how this would work - which ran to several pages. He also produced this diagram.

It's currently fashionable in agile circles to do Value Stream Mapping as a way to uncover waste in a software development process. I think of this as a request stream map because it similarly takes a request and shows how it moves through the layers allowing us to visualize what's going on and think about the cost and value of the layers.

Layering is an essential tool for building software applications. But like most essential things in life, excess can be almost as much of a problem as too little. A visualization like this (or the multiple pages of code) can help you find where "just enough" is.

One hazard, however. If you do need to transform data from one form to another, it's usually better to a few little transformations than one big transformation. You want to avoid unnecessary transformations not compress the ones you need.

Sam Newman: Just won on ebay: Nikon FM3a. Might have to freeze myself and get a friend to thaw me out for its...

Wed, 01 Jul 2009 18:48:18 +0000

Just won on ebay: Nikon FM3a. Might have to freeze myself and get a friend to thaw me out for its arrival date. Flawless plan.

Sam Newman: Just read: #D-Day by Antony Beevor. By turns harrowing, entertaining and insightful. Recommended.

Wed, 01 Jul 2009 18:48:18 +0000

Just read: #D-Day by Antony Beevor. By turns harrowing, entertaining and insightful. Recommended.

Sam Newman: Now reading: Agile Estimation and Planning by Mike Cohn. Off to a dry start - I hope it picks...

Wed, 01 Jul 2009 18:48:18 +0000

Now reading: Agile Estimation and Planning by Mike Cohn. Off to a dry start - I hope it picks up…

John Hume: A Quick Look at Azure, Microsoft's Cloud

noreply@blogger.com (John Hume) — Wed, 01 Jul 2009 15:17:02 +0000

Last week at a ThoughtWorks home-office gathering in New York we had a presentation by one of Microsoft's Architecture Evangelists about their in-the-works cloud offering, Azure. I have to say I was pretty impressed. (If not for the whole Windows thing, I'd probably be trying it out instead of writing about it right now. If you aren't as bothered by Windows as I am, you can register and try it yourself.) Here are some fairly random impressions, mostly based on comparison to the two other best-known cloud contenders.

Azure conceptually leans more towards Google AppEngine's "scale me up; scale me down; hide the details" approach than Amazon EC2's easy-to-manage-infrastructure-as-a-service model, though it currently sits somewhere between them. There's not (yet, at least) any automatic scaling up and down, which surprised me. However what you manually configure is just the number of instances of each application, which is far simpler than manually spinning up new instances of EC2 images and getting them into your cluster (or what-have-you). (Although if you've set up your infrastructure right, your EC2 scaling activities could be just as simple. EC2 just gives you the freedom to do it badly.)

There are two-and-a-half types of application: web apps, worker apps, and web+worker apps. The worker concept is very cool and not something you can do with AppEngine. (The closest thing is their support for scheduled tasks, but these are limited to scheduled URL hits no more often than once a minute with the same restrictions as a normal web request.) Apps all run on something pretty close to the normal .NET framework (minus normal file system access and presumably some other areas--I forgot to ask about threading.)

There are three kinds of storage.

A file-system-like BLOB storage API, which involves saving individual blocks and then telling the system in what order to stitch them together. It seemed clunky, but I could see writing a simple abstraction over it for small stuff (where you'd usually only need one block) and for large stuff maybe being able to take advantage of already having some blocks uploaded when something gets interrupted and can resume later.
A BigTable-like "table" storage API. It isn't as rich as AppEngine's BigTable-backed Datastore in that entity properties are restricted to very simple types (no lists, for example) and there's no built-in support for relationships of any kind (even the simple hierarchy of Datastore). (Note there's nothing to stop you from storing keys of other entities as property values and rolling your own awkward relationships.) Similar to Datastore on AppEngine, you have to think about how you want your data partitioned and make some under-informed guesses about slow queries across multiple partitions vs parallel queries each in their own partition. (On Azure, the key for each entity includes an app-generated token naming the partition in which it should reside, which is a bit simpler than Datastore's parentage-based entity groups.) The story on transactions (being limited to a single entity group) seems to be similar as well. There's also currently no indexing by anything other than key, though it sounded like they'll allow you to add at least one additional index per table before they go proper-live.
Most interestingly, damn-near plain old SQL Server storage in the form of SQL Data Services, with just about every feature of MS SQL Server any normal application would use (except distributed transactions). I believe this is a separate service that you'll need to sign up for and eventually pay for separately. Regardless, it's a key element of the offering, because it provides a huge boost to the suggestion that you can migrate existing .NET applications into the cloud.

There are also persistent queues, which web apps can use to communicate to worker apps asynchronously. (I guess this qualifies as a fourth kind of storage.) They feature guaranteed delivery (with possible repeated delivery) of messages. This is very cool and not something AppEngine offers. (Even if you rolled your own queue-ish thing in datastore, there's no facility to consume messages intelligently.)

There are REST APIs (and SOAP as well, I think) for just about everything, so you have a lot of options for architectures (using heterogeneous technologies) that run partly on-premisis and partly in the Azure cloud.

There are a bunch of other services you can hook into, including ".NET Services" (low-level technical stuff including a Service Bus, Access Control, and a Workflow Service) and "Live Services" (higher-level end-user-meaningful services such as Contacts, Windows Live ID, and assorted Bing stuff).

There's also, surprisingly, the ability to run unmanaged code in your .NET application, AND some form of support for non-.NET stuff via IIS's FastCGI support.

Anyway, it looks like it'll be a compelling offering, competitive with EC2-hosted Windows, and thanks to SQL Data Services much closer to allowing you to put a normal .NET web app into the cloud than AppEngine can do for Java web apps. For any organization willing to trust Microsoft with its infrastructure, Azure is a good reason to be excited. For the rest of the world, carefully executed EC2 still looks like the best bet for the Enterprise in the cloud.

John Hume: Clojure on Google AppEngine

noreply@blogger.com (John Hume) — Wed, 01 Jul 2009 15:15:33 +0000

The release of Java support for Google AppEngine means more than just Java: it means lots of cool JVM languages! My personal favorite is Clojure, so when I and several colleagues got an opportunity to try out a pre-release version (thanks to a partnership between Google and ThoughtWorks) I immediately started trying it out. It turns out to be pretty easy and pretty great.

I'm going to walk you through using Clojure on Google AppEngine by stepping you through the creation of a Clojure/Compojure version of the Guestbook application from the Getting Started section of the AppEngine docs. As we go I'll introduce some bits of appengine-clj, a library I extracted along the way to keep anything that felt like boiler-plate code out of my app.

First you'll need to sign up (if you haven't already) and download the SDK.

Creating Your Application

To start, create a project directory and the basic project directory structure documented in the getting started section of Google's documentation. Here's an overview of what to create and what will go where.

project-root/
  src/
    [Clojure source code]
  war/
    [static files]
    WEB-INF/
      [config files]
      classes/
        [compiled classes from ../../src]
      lib/
        [jar dependencies]

Copy clojure.jar, clojure-contrib.jar, and compojure.jar into WEB-INF/lib. You'll also need to add appengine-api-XXX.jar from the SDK. I'm using fairly recent trunk versions of Clojure, clojure-contrib, and Compojure. (The easiest way to make sure you have compatible versions of these is to pull the latest Compojure source from github and then download their deps.zip and build Compojure against that.) If you want to use appengine-clj you can build it yourself from source or grab a prebuilt jar from the downloads section on Github.

Hello, World!

The entry point to your application will be a servlet class. Create a Clojure source file in your src directory and include a :gen-class directive to extend HttpServlet. Use Compojure's defroutes and defservice to create a HelloWorld.

src/guestbook/servlet.clj

(ns guestbook.servlet
  (:gen-class :extends javax.servlet.http.HttpServlet)
  (:use compojure.http compojure.html))

(defroutes guestbook-app
  (GET "/"
    (html [:h1 "Hello, World!"])))

(defservice guestbook-app)

defroutes above creates a routing function that will respond to an HTML GET for the path "/", and the html function converts vectors to a string of HTML. If you're not familiar with Compojure, start here on the Compojure wiki.

Next create a web.xml with a servlet-mapping sending /* to your servlet class. Since Compojure handles URL routing, your application will have just this one mapping.

war/WEB-INF/web.xml

<?xml version="1.0" encoding="ISO-8859-1"?>
<web-app 
    xmlns="http://java.sun.com/xml/ns/javaee" 
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd"
    version="2.5">
  <display-name>Clojure Guestbook</display-name>
  <servlet>
    <servlet-name>gb</servlet-name>
    <servlet-class>guestbook.servlet</servlet-class>
  </servlet>
  <servlet-mapping>
    <servlet-name>gb</servlet-name>
    <url-pattern>/*</url-pattern>
  </servlet-mapping>
</web-app>

Create an appengine-web.xml, putting your application ID into the application element.

war/WEB-INF/appengine-web.xml

<appengine-web-app xmlns="http://appengine.google.com/ns/1.0">
  <application>[your application ID]</application>
  <version>1</version>
  <static-files />
  <resource-files />
</appengine-web-app>

Create an ant build.xml file that compiles your application to the classes directory.

build.xml

<project name="guestbook-clj" basedir="." default="compile">
  ... here you'll need to define project.classpath ...
  ... see GitHub for the full working example file ...
  <target name="compile" depends="...">
    <java classname="clojure.lang.Compile" classpathref="project.classpath" failonerror="true">
      <classpath path="${src.dir}" />
      <sysproperty key="clojure.compile.path" value="${classes.dir}" />
      <arg value="guestbook.servlet" />
    </java>
  </target>
</project>

I've left out some of the details, but you can find a full working version on Github.

At this point you should be able to run your "Hello, World" iteration of the application locally using the development appserver and deploy to appspot using the appcfg executable (or an Ant task if you prefer). Now it's just a matter of building your application using the tools Google AppEngine, Clojure, and Compojure make available to you.

The User Service

AppEngine has a simple API for dealing with user accounts. Let's greet logged in users by name. You'll need to import com.google.appengine.api.users.UserServiceFactory.

(ns ...
  (:import
    (com.google.appengine.api.users UserServiceFactory)))

...
  (GET "/"
    (let [user-service (UserServiceFactory/getUserService)
          user (.getCurrentUser user-service)]
      (html [:h1 "Hello, " (if user (.getNickname user) "World") "!"]))))

But we have to let users log in to see this work. The UserService also exposes methods for creating login and logout URLs.

...
  (GET "/"
    (let [user-service (UserServiceFactory/getUserService)
          user (.getCurrentUser user-service)]
      (html
        [:h1 "Hello, " (if user (.getNickname user) "World") "!"]
        [:p (link-to (.createLoginURL user-service "/") "sign in")]
        [:p (link-to (.createLogoutURL user-service "/") "sign out")]))))

Now you should be able to log into your application and be greeted by name. On the dev appserver, the login page will let you provide any username and check a box to indicate whether you should be logged in as an administrator for your application. On the appspot servers, you'll get a proper-looking Google Accounts login page. The argument to createLoginURL and createLogoutURL is the path or URL the user should be redirected to after logging in or out.

I've extracted the basic user-lookup calls into a Ring middleware function and put it into the appengine-clj.users namespace in appengine-clj. Here's what our servlet looks like using that.

(ns guestbook.servlet
  ... you no longer need to import UserServiceFactory ...
  (:require
    [appengine-clj.users :as users]))

(defroutes guestbook-app
  (GET "/"
    (let [user-info (request :appengine-clj/user-info)
          user (user-info :user)]
      (html
        [:h1 "Hello, " (if user (.getNickname user) "World") "!"]
        [:p (link-to (.createLoginURL (user-info :user-service) "/") "sign in")]
        [:p (link-to (.createLogoutURL (user-info :user-service) "/") "sign out")]))))

(defservice (users/wrap-with-user-info guestbook-app))

It's about the same amount of code, it just looks a little more clojurey now. (I'm sorry: that's not a word.)

Datastore

Next let's collect guestbook entries and put them in the Datastore. AppEngine for Java has support for a couple of standard Java persistence APIs, JDO and JPA. But we'll use the lower-level datastore API, which seems a better fit for a dynamic language like Clojure (not to mention it doesn't require us to implement Java classes to persist).

The Java API for datastore is pretty simple, but conceptually it's different enough from a SQL database that it definitely takes some getting used to. (I for one am still figuring it out.)

Here's the sixty-second rundown of the bare essentials. The basic unit of persistence is the Entity, which has a Map of String-keyed properties. An Entity has a kind, which is a string denoting the type. (But keep in mind there's no schema here, so you can give any entity any properties.) An Entity is identified by a Key which is something more than a normal DBMS identifier because it can hold an Entity's association with a parent Entity. Besides using the Key, you can retrieve Entities with a Query, which searches either a single kind of entity or descendent entities of a single ancestor (or both, depending on which constructor you use) and can apply simple filtering and sorting.

The natural Clojurized form of an Entity seemed to be a map, so what I've started pulling out into appengine-clj.datastore is functions that allow Clojure code to work with an immutable map of keyword-keyed properties (plus :key and :kind) and have the library take care of translating into Entity objects. Currently there are just create and find methods, since that was all the basic guestbook needed. (But you know that Internet. I'll need a delete function before the week is out.)

Using appengine-clj.datastore, functions for creating and retrieving guestbook greetings are extremely simple.

(ns guestbook.greetings
  (:require [appengine-clj.datastore :as ds])
  (:import (com.google.appengine.api.datastore Query)))

(defn create [content author]
  (ds/create {:kind "Greeting" :author author :content content :date (java.util.Date.)}))

(defn find-all []
  (ds/find-all (doto (Query. "Greeting") (.addSort "date"))))

Note the creation of a com.google.appengine.api.datastore.Query (in find-all) that pulls back all Greetings and orders them by date. I've considered a couple of approaches for cleaning up creation of a query from Clojure, but I haven't decided between something that looks fairly idiomatic vs something that reads just like GQL. For the time being I'm sticking with the Java-interop style since even that is nicely terse and readable. Take a look at the tests for appengine-clj.datastore for more examples.

Speaking of which, this is a good time to mention that writing tests for datastore code is easy with appengine-clj.test-utils, which provides functions to set up an in-memory datastore. The dstest macro used there creates a fresh datastore for each test. If you're using a different testing framework or prefer different scoping, you can call ds-setup and ds-teardown yourself. (Do keep in mind that this is the development version of datastore, so we'll all need to keep an eye out for differences between that and the real Datastore service.)

HTML and Form Handling

Now that we've got our persistence straight (and tested), let's create a UI so users can sign the guestbook. At this point it's just plain Compojure code. We'll create one route to show the guestbook and a form to enter a greeting at "/" and another route for saving the greeting with a POST to "/sign".

Here's our function for signing the guestbook.

(defn sign-guestbook [params user]
  (greetings/create (params :content) (if user (.getNickname user)))
  (redirect-to "/"))

You can see there's very little to it. It takes the request parameters and a user, calls our greetings/create function, and redirects back to the guestbook.

The function for showing the guestbook is quite a bit more to swallow, since it includes our entire user interface.

(defn show-guestbook [{:keys [user user-service]}]
  (let [all-greetings (greetings/find-all)]
    (html [:html [:head [:title "Guestbook"]]
      [:body
        (if user
          [:p "Hello, " (.getNickname user) "! (You can "
            (link-to (.createLogoutURL user-service "/") "sign out")
            ".)"]
          [:p "Hello! (You can "
            (link-to (.createLoginURL user-service "/") "sign in")
            " to include your name with your greeting when you post.)"])
        (if (empty? all-greetings)
          [:p "The guestbook has no messages."]
          (map (fn [greeting]
            [:div
              [:p (if (greeting :author) [:strong (greeting :author)] "An anonymous guest") " wrote:"]
              [:blockquote (h (greeting :content))]])
            all-greetings))
        (form-to [POST "/sign"]
          [:div (text-area "content" "")]
          [:div (submit-button "Post Greeting")])]])))

It takes the user-info map, which it destructures to grab the user and UserService. It calls our greetings/find-all function to get the items to show and then uses Compojure's html helpers to create the document. For any real application you'd want to break the view down into smaller pieces to avoid such a huge nested chunk of vectors (or consider using another templating library like Enlive), but for this example I think it's easier to understand what's going on with the whole page in one function.

Finally here are the routes that wire it all together.

(defroutes guestbook-app
  (POST "/sign"
    (sign-guestbook params ((request :appengine-clj/user-info) :user)))
  (GET "/"
    (show-guestbook (request :appengine-clj/user-info))))

Here I'm using the :appengine-clj/user-info map that's been assoc'd to the request by the Ring middleware.

See the entire servlet file on GitHub, including some enhancements for styling and to see some other code to exercise Clojure features on AppEngine.

The Big Caveat

Two unusual aspects of the Google AppEngine environment create pretty major constraints on your ability to write idiomatic Clojure.

First, an AppEngine application runs in a security context that doesn't permit spawning threads, so you won't be able to use Agents, the clojure.parallel library, or Futures.

Second, one of the most exciting features of AppEngine is that your application will be deployed on Google's huge infrastructure, dynamically changing its footprint depending on demand. That means you'll potentially be running on many JVMs at once. Unfortunately this is a strange fit for Clojure's concurrency features, which are most useful when you have precise control over what lives on what JVM (and simplest when everything runs on one JVM). Since shared references (Vars, Refs, and Atoms) are shared only within a single JVM, they are not suitable for many of their typical uses when running on AppEngine. You should still use Clojure's atomic references (and their associated means of modification) for any state that it makes sense to keep global per-JVM, since there may be multiple threads serving requests in one JVM. But remember JVMs will come and go during the lifetime of your application, so anything truly global should go in the Datastore or Memcache.

More to Come

I'll try and expand on this in the future with more write-ups, including a discussion of special handling for static files (which as of the version of the SDK I'm using works great on the appspot servers even with a /* servlet mapping but not on the local dev appserver, where servlet mappings win out over static files).
If you'll sign my silly little guestbook on the appspot servers, I'd like to publish information on how many requests I got and how they performed.
Google also provides Java APIs for caching, image manipulation, making HTTP requests, and email. I haven't even scratched the surface of those yet.
With fresh support in AppEngine for scheduled tasks and upcoming support for task queues, there's more Clojure fun to be had.

Enjoy!

Marc McNeill: Customer value proposition model

Wed, 01 Jul 2009 09:07:43 +0000

There may be a niche in the market, but is there a market for the niche?

How do you create a successful proposition? If the answer was obvious there wouldn’t be so many failures out there in the market place.

It is easy to commence on a journey of product development with a hunch and clearly there is no substitute for validating ideas in the flesh. That something at ThoughtWorks we do; helping clients test and learn, rapidly building ideas into tangibles that can be piloted at low cost and low risk before investing in significant build and spend. However, sometimes a little more rigour is required before you commit to commencing a project in earnest.

That rigour needs to be focused. What often happens is this rigour turns into a research phase that turns into a project itself. It need not be this way. There are certain things you can do, certain questions to ask as you set out on the journey of creating a new, compelling customer proposition. What follows then is a strawman model to help test potential propositions before moving forward with them. There are three components to the model, the customer, the environmental context and the organisation or company.

All too often propositions are rooted in the organisation. They make assumptions about the demand or usage. This model attempts to broaden the analysis and focus upon the customer and the why the proposition will be attractive to them. The model supports questions that may be asked to help shape thinking, test hypotheses and validate thinking.

I do not propose that this should become a major research exercise (for example market sizing is a huge effort in itself), rather a tool for asking the right questions, and if the answers are hard to come by, maybe that suggests more thought is required in refining the proposition.

So here goes, a model that provides a framework for considering new customer value propositions. It’s just an initial idea and I’d welcome feedback and suggestions.

Customer

Before you get too carried away with the proposition, a good starting point would be the customer. Who are they and what do they do. Let’s remember that your customer is not everybody. Your proposition in unlikely to be appealing 24/7. The challenge is to segment your target market and identify the triggers for action.

The persona: Who do?

Personas are a useful tool for bringing the customer to life. Much has been written about them, but they are a useful tool for extracting broad data into specific stories that describe individuals. Realise that it is unlikely you will design for everybody. Start with the market that you are targeting, how large is it and what is its propensity to spend? Then within that target market segment the target customer base into different profile customers (personas). You need to understand which persona, which customer profile is most important – prioritise them and focus on the highest value. This may mean deciding between high volume, low margin mass market and low volume, high margin niche appeal. This decision needs to be made as early as possible to ensure the proposition remains focused and doesn’t try to be all things to all people, satisfying none.

Values, needs, wants and desires

People are not empty vessels waiting to consume and be filled with your proposition. Their behaviour is driven by their values, needs, wants and desire. These may be fundamentally rational (to satisfy a basic human goal) or emotional (to demonstrate status). They are cultural and time based. Thinking in these terms helps you understand how the proposition will appeal to the customer at different levels. Let’s take an example of this; a new mobile phone.

Before we think about what the product must do, what are the values that the persona associates with the phone. Is our target market a technophile or a technophobe? Jan Chipchase who works for Nokia includes ethnography in his research to understand how people use their phones; women carry them in their handbags, men in their pockets or their belts.

The basic need that the phone must meet to satisfy the customer, she must be able to make and receive calls. If the product is unable to meet these needs it is not fit for purpose and the phone proposition will inevitably fail.

Just making phone calls meets the need but there are additional wants that should be satisfied for the product to be more compelling. It’s a hassle to remember the number of every person she rings, the customer wants to be able to store numbers and see the number of the person who is calling.

Having the ability to see a photograph of her daughter as a screen saver on her phone is neither a need not a want. The phone is useful and usable without that. But the customer desires to personalise her phone by having a picture of her daughter on it. Desirability is the key differentiator of the iPhone. It doesn’t need to compete on features, it is a cool device that people talk about. And here is a key decision you need to make on your proposition journey. Are you looking to compete on parity or whether you want to make a difference.

Questions

What is the basic need that the proposition is trying to fulfil?
What counts as hygiene?
What does the customer need to be satisfied?
What does the customer want in addition to being just satisfied
What do other competive products do to maintain feature parity (if you feel you really need to compete on features alone – bad move!)
Few people would argue they don’t want simplicity and clarity in their interactions with products. How could your product to make life easier for the customer?
What will make the customer feel good in themselves about owning the product?
What other products are “cool” or desirable to your target market. How can you leverage the essence of those products?

Context

So now we are beginning to understand who the customer is, it is time to nest the proposition in terms of their context. The old maxim that a half drunk bottle of water in a desert is worth its weight in gold, but on the streets of a city is worthless trash, should be remembered. Even the best of propositions will deliver little value if they not only consider the customer, but also the context in which they apply: time, demand and usage.

Trigger

So the next step in the model is to ask why, when and how will the customer be attracted to the proposition. What is the trigger that drives the customer to move from awareness (assuming you have that) to action? There is no point in a financial services company trying to sell me a car loan if I am wealthy enough to own my own car, or I do not drive. Understand what triggers the customer to be interested in the proposition, when and why this happens. How can your proposition be at front of mind when the trigger is set.

Questions

What lifestyle / lifestage events will trigger?
Internal events personal to the customer; leaving school, getting a first job, getting married, moving house, retiring etc
External events that they have no control over (think about sports sponsorship and tying a proposition to that sport, or tying a proposition to a celebrity e.g. Michael Jackson..)

Environment

It is very unlikely that the proposition will be wholly unique. What is the competitive landscape, what noise will it need to be heard above to capture the consumers attention. Whilst you may review the immediate competitors to see where threats and opportunities lie, what can you learn from other, unrelated products or domains? How can you fuse together concepts from outside your immediate focus to bring new innovation to your product? Scenario planning may come in useful, playing out different outcomes for different timelines other than that which you plan for.

Questions

What is the competitive landscape?
What can you learn about similar but unrelated propositions?
Have you considered the political, environmental social and technical influences using the old PEST analysis?
Have you considered different scenarios and how your proposition would play out under them; what unplanned disruptors could get in the way, or how could your proposition done differently disrupt the market?

The experience engine

Enough of the customer and externalities, what will the proposition look like and why will the target customer go with it? There are three engines within the organisation that drive the proposition, the experience, delivery and value engines. So…

Utility

To be any good, the product has got to offer basic utility. It has to do what it says it is going to do. Sadly, too many products and customer propositions end there. A utility product will match the consumers needs. This is where most enterprise software sits…

What are the key customer needs that the proposition must fulfil?
What is the basic core functionality that must be met, what are the features that must be offered to gain traction in the market place?
What features that are typical on competitor products that we could do without?

Quality

I could call this next box usability (as this follows the UXD model) but I think it goes beyond just usability. What is the quality of not only the immediate interface, but also with the supporting functions? For example, if you have a call centre to back up the proposition, how many layers of IVR are you forced through?

Have you considered usability?
Is the packaging aesthetically pleasing?
The “happy path” customer journey may be well framed, but what about the “sad path”? What about when things go wrong, what about when customers don’t act in the way you expect of predict them to act?

Brand

It is easy to get carried away with a new idea before thinking about what it means to the brand. Typically there will be a strategic roadmap and whilst the proposition may be attractive it may not fit into where the brand is going.

Is the proposition complementary to the overall brand direction or does it require a new brand and identity?
Does the proposition support / leverage the brand?
Does the brand already ‘do it’ under another guise (are you reinventing a wheel that has already been tried somewhere, sometime in the organisation’s history?)
How will it be marketed?

Community

Finally, what is the ‘buzz’ that the proposition will create, what will get people talking and sharing it and how will you create this buzz.

Is there a social network component built in that gets people talking and connected? How will it get people talking in external networks?
What will cause people to recommend it to others?
How can customers become part of its evolution?
What of the proposition will get people passionate, what will drive them away?

Delivery engine

People

A successful proposition needs not only a talented, passionate and committed team to deliver it to market, it also needs a similar team to run it and support it when it is live. It is a common failing for a rogue “skunkworks” team to emerge in an organisation and develop what appears a compelling proposition, only to have it knocked back and closed down by the “Business as Usual” processes inherent in the organisation

Who do you need to make the proposition successful? What is the team?
Who will create the proposition and who will lead it? Is it IT led or business led?
What are the cross-organisational boundaries that the proposition crosses and how will these be eliminated?
Who will take ownership of the proposition once it crosses over into the market?

Process

What are the processes that will be required to sustain the proposition?
If the proposition will require changes to the organisation, how will they be managed, communicated and rolled out?
How will the proposition be supported once it is let loose in the market?
How will it be communicated to customers?
How will you create new sales – sales force.

Technology

What is the technology that will underpin the proposition?
Is it possible to test the ideas using rapid languages such as Ruby on Rails before committing it to the enterprise Java stack?
What integration is really necessary and what can be worked around?
How can you deliver a beta version in the shortest period of time?
How will you avoid heavyweight frameworks and develop incrementally to deliver value early and often?
How performant and scalable must the innovation be?

Value Engine

At its most simplistic, how much will the proposition cost and how much revenue will it generate? Does it offer cost saving opportunities? Are there intangible benefits that will be accrued? Ultimately is it a viable proposition that is worth pursuing, or will the cost to develop and run outweigh the value it will add? Building out a financial model can take time, in the first instance this should be a napkin analysis, a wake-up call to make sure there is value in the proposition before too much time is invested in it.

Cost

Every day someone is working on the proposition it is costing you money. The quicker you can get something to market the faster you will start seeing a return on your investment, similarly the sooner you can “get something out there”, “test and learn” the sooner you can kill a proposition that does not fulfill its promise.

How quickly can you get a beta to market?
How many people, how many days?
What will the cost be to develop the infrastructure?
Do you have the skills in house or will you need to go external?

Benefit / Revenue

At its most crude, how will the proposition make money, but there may be more to what we wish to achieve. Is the proposition actually going to cut costs, a result of regulatory pressures or a CSR initiative?

What are the benefits that will be accrued – both tangible (e.g. financial) and intangible (e.g. social, environmental etc)

If you are selling units are you going for high volume low margin or low volume high margin?
If it an on-line proposition “advertising” is often seen as the source of revenue.

There are two additional components to the model…

Implementation

Having a compelling proposition is one thing, it is another to successfully communicate it and roll it out to target customers.

In a crowded market place, how will the proposition stand out?
What are the brand values it will communicate?
What is the story that customers will hear and how will they hear that story?
How will customers interact with the proposition, what channels will you use to take it to market?
What is the roll out strategy?

Retain and grow

Winning customers is only the first step. A successful proposition will maintain a long-term relationship with its profitable customers, maintaining the warmth they have to the original proposition and cross-selling and up-selling new ones.

How will you retain them and turn them into repeat customers and passionate advocates of the proposition?
How will the proposition grow lifetime customer value?
What can be cross-sold or up-sold?
What can you bundle?
How will the proposition deal with churn?

OK, so it’s not a perfect model and by no means complete. There’s some duplication in the thinking and many questions missing, but as any model it can be used to guide and prompt thinking and ensure there are no elephants left in the room when the first line of code gets cut. I’d welcome any comments on its usefulness, utility and direction.

Ted Neward: Review: "Iron Python in Action" by Michael Foord and Christian Muirhead

Wed, 01 Jul 2009 09:00:14 +0000

OK, OK, I admit it. Maybe significant whitespace isn't all bad. (But don't let me ever catch you quoting me say that.)

The reason for my (maybe) shift in thinking? Manning Publications sent me a copy of Iron Python in Action, and I have to say, I like the book and its approach. Getting me to like Python as a primary language for development will probably take more than just one book can give, but... *shrug* Who knows?

Bear in mind, I have plenty of reasons to like IronPython (Microsoft's Python implementation for the .NET environment):

A good friend of mine, Harry Pierson (aka @DevHawk), is the PM on the IPy project, and I'm generally prejudiced in favor of those things that people I know and respect.
I'm generally a fan of dynamic languages, particularly those that let you do strange and twisted things to the type system and its instances at runtime. (Yes, I'm looking at you, ECMAScript...)
I spent some quality time with IronPython Studio last year while researching a Visual Studio Extensibility "Deep Dive" paper.
I've known Jim Hugunin (the creator of IronPython, and Jython before that) for some years, ever since his days working on AspectJ, and he's one of those scary-smart guys that, despite knowing they're scary-smart, still render me stunned when I listen to them.
I'm a huge fan of the DLR. It's like having Parrot, but without having to wait a decade (give or take).

But, just to counterbalance the scales, I have plenty of good reasons to dislike IronPython, too:

Significant whitespace.
The "There's only one way to do it" oath that Pythonistas seem to hold as religion. (Somebody told me that building C-Python—the original implementation—only works for you if you swear a holy oath to The One True Way on the One True Way Bible. Needless to say, I believe them, and have never tried to build C-Python from sources as a result.)
Significant whitespace.
Uh.... did I mention significant whitespace yet?

I admit, it was with some hesitation that I cracked open the book. Actually, to be honest, I was really ready to just take out all my dislike of significant whitespace and pour it into a heated, vitriolic diatribe on everything that was just wrong with Python.

And...?

Well, OK, I admit it. Maybe significant whitespace isn't all bad.

But this is a review of the book, not the technology. So, on we go.

What I liked about the book

The focus is on both .NET and Python, and doesn't try to short-change either the "Python"-ness or the ".NET'-ness by trying to be a "Python book (that happens to run on .NET)" or a ".NET book (that happens to use Python for code samples)". The authors, I think, did a very good job of balancing the two, making this the book to get if you're in that area on the Venn diagram where "Python" overlaps with ".NET".
Part 2, "Core development techniques", starts down the "feed you the Python Kool-Ade" pretty quickly, heading straight into Chapter 4 ("Writing an application and design patterns with IronPython") without much of a pause for breath. The authors get into duck typing, protocols, and Model-View-Controller within the first four pages, and begin working on a running example to highlight some of the ideas. (Interestingly enough, they also take a few moments to point out that IronPython on Mono works, and include a couple of screen shots to that effect as we go, though I personally wonder just how many people are really going down this path.) I like the no-holds-barred, show-you-the-code style, but only because they also take time throughout the prose to talk about some of the concepts at work underneath and laced throughout the code. "Show me then tell me" is a time-honored tradition, but too many authors forget the "tell me" part and stop with code. These guys do a good job of following through.
The chapters in Part 3, "IronPython and advanced .NET", form an interesting collection of how IronPython can fit into the rest of the .NET stack, demonstrating how to use IronPython with WPF, ASP.NET, and IronPython's crowning glory, Silverlight. If you're into front-end stuff, this is the section where I think you're going to have the most fun.
The chapters in Part 4, "Reaching out with IronPython", is I think the most important part of the book, showing how to extend IronPython (chapter 14) with C#/VB extensions (similar to how a C-Python developer would extend Python by writing C code, but much much simpler) and the opposite—how to embed IronPython inside of existing C#/VB applications (chapter 15), which is really an exercise in using the DLR Hosting APIs. While the discussion in chapter 15 is good, I wish it'd had a bit more thorough discussion of how the DLR could be hosted regardless of the scripting language, though I admit that's pretty beyond the scope of this book (which is focused, after all, entirely on IronPython, and as a result should stay focused on how to host IPy).

What I found "Meh" about the book

Part 1 ("A new language for .NET", "Introduction to Python", and ".NET objects and IronPythong") does a good job of bringing the rank beginner up to speed, getting some basic Python ideas across in the same breath that they bring .NET home. The only problem is, it only works well if you're neither a Python programmer nor a .NET programmer. Chapter 1, for example, does a sort of Cannonball-into-the-pool kind of dive into Python, but dives equally into the "Iron" parts as it does the "Python" parts. If you're either a Pythonista or a .NETter, I suspect you're going to be tempted to flip pages pretty quickly, and (I suspect) miss a few things. Chapter 2 is all about Python (meaning .NETters will probably spend some time here), but it certainly doesn't feel like an exhaustive reference, nor does Chapter 3 stand as an exhaustive discussion about all things .NET, either. I almost wish all three chapters had been collapsed into one—suffice it to say, I don't feel like I know the Python language, and don't feel like this book could be my Python reference next to me as I learn it, and I know that it's not a great .NET reference, either. Fortunately, the goal of these three chapters feels pretty clearly to be "Teach you just enough to make you dangerous (and able to understand the rest of the book)", and once we hit Part 2, rubber meets road pretty quickly.
By the time you hit Chapter 7, less than halfway through the book, the authors have created a fairly nice, if simplistic, application for later dissection, but it's not until you hit Chapter 7 that they begin to start unit-testing, even though they insist (on page 17) that "Dynamic language programmers are often proponents of strong testing rather than strong typing" (a quote they attribute to Bruce Eckel, though I'm relatively certain I heard Dave Thomas and Neal Ford say it with respect to Ruby, long before Eckel started "Thinking in Python... or Flex... or whatever"). If unit-testing is that important, why wait three chapters into the application's development before writing a single unit-test? This doesn't jibe with me, somehow.
If you're into back-end stuff, chapter 12 on "Databases and web services" is pretty bland. The fact that the two are combined into a single chapter is indicative, all by itself, of how deep or intensive the coverage goes, and there's zero mention of anything beyond basic ADO.NET. The coverage on web services covers REST relatively well, but there's zero coverage of WCF, and the whole of SOAP-based services is all of four or five pages. And Workflow? Doesn't exist, isn't even mentioned (except for an appearance in a table, "The major new APIs of .NET 3.0"). Yikes.

What I actively disliked about the book

Actually, not much. Manning did their usual superb job of arrowed callouts to point out particular concepts in the code listings, the copyediting is professional (meaning there's no obvious typos or misspellings that just break up the flow of prose, something that not all publishers seem to take seriously), and the graphics flow nicely alongside the prose, not dominating the page but accentuating it.

In fact, about the only thing I'd care to criticize is the huge number of footnotes, particularly in the first chapter. (By page 20 in the book, there have already been 30 footnotes.) When you have three footnotes per page, on average (and sometimes more), it does tend to distract, at least to me it does. It feels like there were ways, for most of them, to inject the idea or concept into the main prose, or leave it out entirely, but that could just be a difference of writing style, too.

Summation

If you're a .NET developer interested in learning/using IronPython on your next project, this is a definite winner. If you're a Python developer looking to see how to break into .NET, I'm not so sure this is your book, but I say that mostly because I'm not a Pythonista and can't really speak to how that mindset will find this as an introduction to the .NET space. My intuition tells me that this would be a good springboard into another book on .NET for the Python programmer, but I'll have to leave that to Pythonistas who've read this book to comment one way or another.

Enterprise consulting, mentoring or instruction. Java, C++, .NET or XML services. 1-day or multi-day workshops available. Contact me for details.

Erik Doernenburg: Making ESB pain visible

Wed, 01 Jul 2009 05:40:39 +0000

At some point last year I was asked to review the architecture of the software behind a large and popular website. The resident architect explained how he had followed a modern approach, decoupling the web front-end from back-end services that provide content. To gain further flexibility he had put the front-end and the services on an ESB, mostly to cater for the possibility to make the content available to other consumers. In short, the architecture diagram looked a lot like many others: nothing to see here, move on.

The diagram above only shows one of the content services, which for the sake of this article is a service that provides contact details for a person.

Based on conversations with the project sponsors I began to suspect that at least the introduction of the ESB was a case of RDD, ie. Resume-Driven Development, development in which key choices are made with only one question in mind: how good does it look on my CV? Talking to the developers I learned that the ESB had introduced “nothing but pain.” But how could something as simple as the architecture in the above diagram cause such pain to the developers? Was this really another case of architect’s dream, developer’s nightmare?

Often a picture says more than a thousand words, but at the same time traditional architecture diagrams simply do not provide enough information. (I’ve argued this before, here and here for example.) The answer to my questions, and the anxious project sponsors’ question whether the project would deliver on time, lay in a different picture.

The picture should not only show the layering of the system in more detail than the architecture diagram but it should also visualise the complexity introduced by the ESB. It’s often possible to produce more detailed diagrams automatically from source code, configuration, or runtime inspection but in this case I couldn’t find a way to do that, which is which I drew the following:

In the centre column this diagram shows the layers involved, from the Struts action that needs to look up details by an ID down to the back-end that provides a “RESTful” service to retrieve the details. The top box contains the front-end layers while the box in the middle represents the ESB. The layers labelled in italics are frameworks or infrastructure software, the rest is project-specific code.

Key to this diagram are obviously the coloured representations of data travelling up and down the stack. Primitive types are grey, Java objects are purple circles, XML documents are shown as turquoise hexagons, and HTTP messages as green blobs. Mule messages, which are Java objects, are shown in dark purple. If one representation is wrapped in another it’s displayed inside the wrapper.

The diagram highlights that it’s a simple piece of data that travels all the way down the stack (actually further until it hits a relational database at some point) and, equally, that the data needed by the action is available in the right form from the service. All that the intermediate layers are doing is wrapping and unwrapping the data in various ways. In fact, out of the four layers of project-specific code three layers do nothing but wrapping. No wonder the developers felt some pain.

As a side note: the diagram also makes it obvious that the wrapping is asymmetric between the WsToRestHandler and the RestServiceWrapperImpl, which makes the point of this layer boundary even more questionable.

This diagram made people more interested in the cost of what had just been a simple, innocuous line in the architecture diagram; which was the point of the exercise. So, what would this diagram look like without the ESB? Easy:

Apart for the obvious simplicity the diagram highlights that every layer has a purpose and there is only one wrapping, which is not undone either. Would it surprise you if I told you that the project went live without the ESB?

Footnotes

1) This article is by no means implying that any of the mentioned technologies are useless or harmful. Their use was harmful in this case.

2) The pattern of wrapping and unwrapping the same data through multiple layers is not new, it was also quite common in early 3-tier architectures. (In fact, some people called their architectures n-tier; I guess to show that the number of tiers was somewhat arbitrary but large.) We had a term to describe this pattern: the wormhole pattern.

3) This project is not an isolated incident of a Wormhole ESB. I have personally seen several in the past years.

Sidu Ponnappa Kariappa Chonira: CruiseControl.rb 1.4.0 released!

noreply@blogger.com (Sidu) — Tue, 30 Jun 2009 20:28:54 +0000

We are happy to announce the release of CruiseControl.rb 1.4.0. This release adds support for three distributed version control systems - Git, Mercurial and Bazaar - in addition to Subversion.

CC.rb remains easy to install, pleasant to use and simple to hack. Since the source has now moved to a git repository, it is easier than ever to fork and contribute. We're looking forward to your pull requests!

Downloads are available from both Rubyforge and Github.

ThoughtWorkers on Open Source: CruiseControl.rb 1.4.0 released!

noreply@blogger.com (Sidu) — Tue, 30 Jun 2009 20:25:43 +0000

Martin Fowler: IllustrativeProgramming

Tue, 30 Jun 2009 19:23:00 +0000

What's the most common programming language in the world?

I'm not sure how you could go about measuring this, but one thing you'd need to do is consider what we mean by programming. My candidate answer considers that the most popular programming language is one used widely by people who do not consider themselves as programmers. This language is Excel, or more generally spreadsheets.

Spreadsheets are easily used for small tasks, but are also used for surprisingly complex and important things. Often I've seen professional programmers gulp when they realize that some vital business function is being run off some spreadsheet that they'd find too complicated to muck with.

In general, we've not had much success with programming languages for these kind of LayProgrammers. Whenever someone talks about some new environment that's going to allow people to specify complex behavior "without programming" I mention COBOL, which was originally designed to get rid of programmers. So it's important to consider what Excel can teach us about programming environments.

One property of spreadsheets, that I think is important, is its ability to fuse the execution of the program together with its definition. When you look at a spreadsheet, the formulae of the spreadsheet are not immediately apparent, instead what you see is the calculated numbers - an illustration of what the program does.

Using examples as a first class element of a programming environment crops up in other places - UI designers also have this. Providing a concrete illustration of the program output helps people understand what the program definition does, so they can more easily reason about behavior.

So why do I feel we need this particular Neologism? Essentially because I think it deserves more thought. We pass by illustrative programming examples without really thinking about them or what makes them special - or even that they are special in some way. We've used illustrative programming for years, but we've not paid enough attention to it. We've not thought enough about what are its essential qualities and what its strengths and weaknesses are.

I've chosen the term "Illustrative Programming" to describe this, partly because "example" is so heavily used (and illustration isn't) but also because the term "illustration" reinforces the explanatory nature of the example execution. Illustrations are meant to help explain a concept by giving you a different way of looking at it - similarly an illustrative execution is there to help you see what your program does as you change it.

When trying to make a concept explicit like this, it's useful to think about the boundary cases. One boundary is the notion of using projections of program information during editing, such as an IDE that shows you the class hierarchy while you are working on the code. In some ways this is similar, as the hierarchy display is continuously updated as you modify the program, but the crucial difference is that the hierarchy can be derived from static information about the program. Illustrative programming requires information from the actual running of the program.

I also see illustrative programming as a concept beyond the classic REPL loop of dynamic languages. REPL loops allow you to explore execution, but they don't make the examples front and center in the way that a spreadsheet does its values. Illustrative programming techniques put the illustration in the foreground of your editing experience. The program retreats to the background, peeping out only when we want to explore a part of the illustration.

I don't think that illustrative programming is all goodness. One problem I've seen with spreadsheets and with GUI designers is that they do a good job of revealing what a program does, but de-emphasizes program structure. As a result complicated spreadsheets and UI panels are often difficult to understand and modify. They are often riven with uncontrolled copy-and-paste programming.

This strikes me as a consequence of the fact that the program is de-emphasized in favor of the illustrations. As a result the programmers don't think to take care of it. We suffer enough from a lack of care of programs even in regular programming, so it's hardly shocking that this occurs with illustrative programs written by lay programmers. But this problem leads us to create programs that quickly become unmaintainable as they grow. The challenge for future illustrative programming environments is to help develop a well structured program behind the illustrations - although the illustrations may also make us rethink what a well structured program is.

The hard part of this may well be the ability to easily create new abstractions. One of my observations of rich client UI software is that they get tangled because the UI builders think only in terms of screens and controls. My experiments here suggest to me that you need to find the right abstractions for you program, which will take a different form. But these abstractions won't be supported by the screen builder as it can only illustrate the abstractions it knows about.

My colleagues Rebecca Parsons and Neal Ford have been spending a lot of time involved in thinking along these lines too. So here's some thoughts that Neal had in an email exchange

I think these tools work best for lay people (thus, your link to LayProgrammers). However, in general, tools like this slow down experienced/power users. When you mention UI panels, the Mac is rife with these types of controls. I spend a great deal of time in Keynote, fiddling with the inspector. At least all those controls are in one place (not like the new ribbon stuff). I would much prefer a markup language I could use to directly define stuff, with macros, snippets, and all the other things I'm accustomed to as a developer.
as these tools grow, they get unwieldy (perhaps because they are ceasing to be domain specific enough?) Look at Word, Excel, and PowerPoint. They had to invent new UI metaphors to expose all the functionality of those tools. APIs in programming languages scale much better, with several orders of magnitude more density before they become hard to navigate.
All the best-practices and tools don't exist there: refactoring, levels of testing, etc. Also, you loose the connection to text, meaning that macro facilities either don't exist or complex one-offs. I think a good comparison that highlights the limitations of Illustrative Programming is the comparison between bash (large, arcane, powerful, quirky) to Automator. I almost never use Automator because it suffers from Dietzler's Law: it's always lacking 10% of what I need. I gladly deal with the crufty surface area of bash because of the more power afforded.
I share your bullishness around these types of tools, but they are a long time from being useful for full-bore Agile development. I hope they mature fast.

--Neal Ford

One of the few people to take illustrative programming seriously is Jonathan Edwards. He's come up with many very imaginative ideas as to what such an environment should look like. His vision of illustrative programming is also closely bound to the notions of projectional editing and controlled copy-and-paste.

The trigger for me in wanting to coin a term here, is the use of illustrative programming by Language Workbenches by people like IntentionalSoftware. These Language Workbenches encourage you to build illustrative DSLs. Using illustration is important in this case since this should help engage lay-programmers, which is one of the aims of using DSLs. The challenge is to do this without falling into the trap of poor program structure.

Ye Zheng: 和新手一起工作

Tue, 30 Jun 2009 15:43:53 +0000

不知道从什么时侯开始，周边的人把我当作一个老手，尽管有些自己不那么情愿，但现实是和我在一起工作的人大多比我工作经验少。

新手之所以为新手，是因为他们的经验较少，所以，不可避免的会犯一些错误。记得别人还在用新手标准要求我时，有一次出差，到了现场，项目负责人要我去安装我们的程序，可我根本就没把我们的程序带来，结果可想而知，项目负责人劈头盖脸的把我骂了一顿。

换我扮演老手的角色，我会尽我所能把项目中的一些不确定问题解决掉，然后，让新手们去解决那些确定性问题，通常，他们的能力应对这样问题会游刃有余。最近一年多的项目里面，涉及到了很多探索新技术的工作，我会从项目中挑出一个最简单的情形用新技术实现出来，这个例子可以帮助我弄清楚技术背后的来龙去脉。之后，我就可以把他解释给其他人。我的同事们都很聪明，理解了基础结构，加上一个现成的例子，他们就完全可以继续进行下面的工作了。

或许单独看起来，我的做法并没有什么特别的地方。刚好最近项目里面有两件点要去探索，我自己做了一个，把另外一个分给另外一个同事。他很快就做完了，而就是这个最简单的例子，我也做了好几天。探索的工作结束，进入到正式的开发工作，其他同事很快就可以接着我的工作继续开展。就是这个最简单的例子，除了了解技术的目的之外，我还写所有相关的脚本，万事俱备，只欠东风。而当我了解另外一个同事的工作时，我发现，他真的只做了一个最简单的例子，了解了一些基本概念。真正把这些内容运用到项目里时，左一个困难，右一个麻烦相继出现，我陪着他一个个克服了这些点，又把脚本预备好，几天之后，才真正进入到可以开发的状态。

有人曾经问过我，你把有趣的工作都做了，把无聊的工作留给别人是不是合适呢？毕竟每个人都希望在项目中成长。前面说过，我的工作是削除项目中的不确定因素。是的，很多人喜欢的就是解决不确定问题带来的快感。但不确定对每个人来说是不一样的。只要最基础的例子跑通了，这项技术的不确定性对我而言就解决了，但实际上，在具体使用这项技术中还会有一些不确定的问题等待解决，解决这样的问题，同样可以提高。

谈到成长，每个人都希望自己能够不断的成长，但是，即便是做同样的项目，每个人得到的机会也是不一样的，这样的差异实际上对应着各人不同的表现。和其他人一起工作的过程中，通过观察，我会把不同的工作交给不同的人来做。像前面说的那个同事，我之所以肯把一个探索的机会交给他，是因为他在之前的工作中表现出的态度和能力，虽然他的探索并不完全令人满意，但是，有了之前的信任，我会告诉他，怎么做才会做得更好。

之所以会想起这个话题，是因为一个同事与我聊起他项目焦头烂额的状态。他在那个项目组中，扮演着和我类似的角色，他感觉自己做得非常累，项目进展也有些问题，于是，我们俩聊起了如何发挥其他人的作用，既让新人感觉自己得到了锻炼，也让自己轻松一些。

《城客》：第一本中文互动杂志！

Akshay Dhavle: The Revolution

noreply@blogger.com (Akshay Dhavle) — Tue, 30 Jun 2009 13:37:48 +0000

Software Development - Art OR Science?

A seemingly clichéd question. Never passed my mind all these years. But let me tell you how I got thinking about this and maybe it'll interest you a bit.

Like I said earlier, I have been following a lot of Lean-Kanban discussions, articles, etc lately. Some such material is Little's Law & WIP limits. Now the moment I saw an equation, I couldn't resist the temptation of trying out some math to see if the size and composition of my current team is optimal. Furthermore, I thought, given a few specifications of the project like domain complexity and technology, could I find the optimum team size and composition?

I hit two forums with this idea and most of the feedback was that there are too many things to consider and difficult things as well like the skills and experience of the people on the team. And I agree that people make most if not all the difference. But that's the problem isn't it?

What's so special about our people? Skills. Experience. Talent even. It's a sign. It's a sign of our industry being immature. I think its in around the occupation stage on this scale.

Enough demand can trigger the Art - Occupation - Industry - Science transitions for any activity. Note that the transition is never 100%. There's Art and Science in everything. The question is whether something is "more of" art OR science.

People keep arguing that Software Development is different. We are not like construction, we are not like assembly line manufacturing, we are not like Product Development either. I used to believe it till a few days back. But the more I think about it the more I feel that these are arguments of a losing population of craftsmen who are finding it increasingly difficult to meet the demand for their craft (which has BTW risen at unusual rates).

Think about the guy somewhere around current Pakistan who created the first piece of leather clothing many hundred years ago. Think about the leather industry right now. Think about the transition. At some point he must be saying "This is different. This needs skills. This needs experience". It took centuries for the transition but it happened. I am sure it can be co-related to the rise in demand for leather products.

The funny thing is, if you look at the list of general characteristics of the Art and Science sides, the first three points on each side don't really fit in with the IT occupation do they? We have loads of unskilled people sitting around producing amazing amounts of useless code all over the world.

So I think the revolution is inevitable. At some point the people who pays us boatloads of money for bad software are going to revolt. We either have to change OR die.

Here's another article roughly talking about similar things. The author anticipated a code market to emerge where reusable components would be bought and sold (which didn't happen OR hasn't happened yet). But the rest of the content is around the same theme as this post.

Mark Needham: F#: What I've learnt so far

Tue, 30 Jun 2009 13:09:35 +0000

I did a presentation of some of the stuff that I've learnt from playing around with F# over the last six months or so at the most recent Alt.NET Sydney meeting.

I've included the slides below but there was also some interesting discussion as well.

One of the questions asked was around how you would deal with code on a real project with regards to structuring it and ensuring that it was maintainable. I'm not actually sure what the answer is to this question as I haven't written any code in F# that's in production but there are certainly applications written n F# that are in production - the main one that I know a bit about is one which Amanda Laucher worked on which she spoke about at the Alt.NET conference in Seattle.
There was some discussion about dynamic v static languages - Phil spoke of not caring about what type something is rather caring about what it does. I pretty much agree with this and I think when using languages which have quite strong type inference such as F# (and more-so Haskell from what I hear) then I think we do move more towards that situation.
Erik raised the point that functional languages aren't the solution for everything and I certainly feel it's niche is probably around operations with heavy data parsing/mining involved. I'm not sure I'd fancy doing an ASP.NET MVC application only in F# although I've seen some WPF code written using F# (unfortunately can't remember where) which looked reasonable so I'm not sure we should write it off just yet.

I've put the code that I walked through in the presentation on bitbucket.

F#: What I've learnt so far

View more presentations from markhneedham.

Richard Durnall: Top 200 Blogs for Developers

Tue, 30 Jun 2009 08:57:17 +0000

I just made Jurgen Appelo’s list of ‘Top 200 Blogs for Developers’. I’m sneaking in there at number 198, apparently down from 150 when Jurgen last published the list. It’s my own fault for not blogging for almost three months!

There are some great blogs in the list that I’m already aware of and a heap more that I’m going to trawl my way through. Let me know if you come across any that are worth a look…

Oh, and the irony of being a non-developer appearing in the list isn’t lost on me either!

Marc McNeill: If you say Log out, log me out

Tue, 30 Jun 2009 08:26:13 +0000

You’ve logged into your on-line banking checked your balance, paid your bills. What do you do now? Click on the logout button?

What do you expect will happen now? Well given that you have actively chosen to log-out (it’s not something you are likely to click on my mistake), you’d expect to do exactly that. Logout. The next screen you get will probably be something that thanks you for on-line banking, with a cross sell for a product or two.

That’s what I assume most customers would expect. So what are Alliance and Leicester thinking about with this screen?

The customer has clicked log-out but they are still logged in?

Why?

“You are still logged in to Internet Banking - before you go have a look at Your offers.”

Excuse me, I logged out, I don’t need to be logged in for you to show me offers.

Worse: “Are you sure you want to log out?”

OF COURSE I WANT TO LOG OUT!!! Why else would I have clicked the link.

Alliance and Leicester fail here in a fundamental usability rule, that of managing the customer’s expectation. In an application where security isn’t paramount this would be an error, in an application where customers expect their action of leaving their secure accounts will do exactly that… but doesn’t, is inexcusable.

Jeff Xiong: Cruise全绿了

Tue, 30 Jun 2009 02:43:53 +0000

好久没有过了也～～Cruise全绿，连IE6的测试也通过了也～～

Cruise是个让人快乐的软件，Mingle是个让人不快的软件，因为Cruise不要你去管它，Mingle总要你拖来拖去 ^_

Ross Pettit: The Case for Restructuring IT

noreply@blogger.com (Ross Pettit) — Tue, 30 Jun 2009 02:26:00 +0000

Business is tough right now, and it’s going to be so for a while. In tough times, you want to be very good at what you do. The more “fighting fit” you are, the more likely you are to survive a challenge.

Unfortunately, IT isn’t all that good at what it does. In fact, on the whole, it’s pretty bad. That means that IT isn’t very well prepared for this downturn.

How bad is it? The research organizations have historically reported a pretty high failure rate of IT projects: about 30% of all IT projects fail outright, while another 30% disappoint their business sponsor (e.g., excessive cost, wrong functionality).¹ On the whole, an IT investment has, at best, a 4 in 10 chance of success.

Companies are already reticent to invest in this climate. IT doesn't offer scared capital a safe haven.

It also suggests that IT is on a trajectory of self-destruction. If we want to look ahead to where IT is headed, we need look no further than present day Detroit.

Photo credit: Ben Wojdyla, The Ruins of Detroit Industry

How did this happen? Consider some of the forces that have shaped the current IT landscape in the past 20 years. The steady growth of IT that was accelerating slightly with the advent of client/server gave rise to explosive growth driven by the combination of internet and Y2K. By the mid-1990's, demand for IT was dramatically outstripping supply. To satiate this demand, IT went in pursuit of scale. To get scale, IT took professional jobs and codified them into industrial tasks, because it’s easier to staff vast numbers of people in highly specialized roles than it is to develop professional capability to solve business problems using technology.

Today, businesses buy, recruit, staff, govern, gatekeep, develop, analyze and test following a model that puts a priority on “big.”

Unfortunately, all the time we’ve been in pursuit of scale, we’ve not been in pursuit of results. Results are assumed. We assume armies of specialists will follow an explicitly defined project plan to produce a solution that is technically sound, functionally complete, and financially satisfactory, all with minimal risk of impairment.

With a 4 in 10 batting average, results cannot be assumed.

By placing a priority to scale, IT mistakes effort for results. We often see success expressed as a function of hours to be invested. It isn’t that simple. Successfully delivering an IT solution is a function of a lot of factors, such as clear communication, effective collaboration and capability; well-informed decision making about technology, functionality and commercial viability throughout the life of a project; flexibility and responsiveness; and ultimately, producing meaningful things for our business partners. These can’t be captured in task orders and forecasts of work effort. They’re lifestyle decisions of how IT goes about its business.

It is time to restructure IT, to move away from an effort-centric industrial model, towards result-centric professional one.

¹ As an example, the 2009 CHAOS report from The Standish Group shows that things haven't changed all that much, reporting 44% of IT projects were challenged (late, over budget, and / or with less than required features and functions) while another 24% failed.

Martin Fowler: Revitalizing Enterprise Software

Mon, 29 Jun 2009 23:14:00 +0000

web site news: AMP, an Australian financial services company, ran an internal conference called Amplify. They asked me to talk about agile software development. I thought about how to make this best fit into the overall flow of the conference, particularly since I expected a significant part of the audience to not be part of IT. I settled on talking about how IT projects can be infrastructural or strategic. This classification alters how you approach the projects, in particular on the way IT and business people should collaborate.

Felix Leipold: CATting multiple files

Mon, 29 Jun 2009 17:28:16 +0000

Quite often I want to pipe the content of multiple files into a command line utility. An example would be to count the lines of sql in my project. This is another case, where xargs comes in handy:

find . -name "*.sql" | xargs cat | wc -l

Felix Leipold: Visualising log files with gnuplot

Mon, 29 Jun 2009 17:17:28 +0000

I recently had the pleasure of supporting a new system throughout its first month of production. This was a good opportunity to refresh my command line skills. As it happened I spent a lot of time looking at log files trying to figure out what happened to the productions system. I figured, that a graphical representation of the events would be nice and started using gnuplot.

First I started out with a bunch of bash scripts, using what your usual unix installation provides, but then I actually came up with some groovy scripts to provide better abstractions. A log file generally looks somewhat like this:

04/01/1970 07:55:13 garbage
04/01/1970 09:27:48 Event 2
04/01/1970 10:01:28 garbage
04/01/1970 10:38:30 garbage
04/01/1970 10:48:36 garbage
04/01/1970 10:51:58 Event 2
04/01/1970 11:03:45 garbage
04/01/1970 11:34:03 Event 1
04/01/1970 12:24:33 garbage
05/01/1970 04:35:50 ERROR

There is a lot of garbage plus some events we might be interested in. It allows to specify events, e.g. by providing a regexp:

 
Event EVENT1 = new RegExEvent("Event 1", ~/Event 1/)
Event EVENT2 = new RegExEvent("Event 2", ~/Event 2/)
Event ERROR = new RegExEvent("Error", ~/ERROR/)

The next step is newing up a TimeLineVisualizer on these events and passing in a stream with the actual log:

 
def logFile = new File("test.log");
 
logFile.withInputStream {InputStream stream ->
  def visualizer = new TimeLineVisualizer([
          EVENT1,
          EVENT2,
          ERROR
  ]);
  visualizer.visualize(stream)
}

If you have the gnuplot binary on your path this will yield something like this:

Also in some cases you would like to know which time of day events are most likely to happen. For producing histograms I created another visualizer (which currently takes only one event).

 
logFile.withInputStream {InputStream stream ->
  def visualizer = new HistogramVisualizer(EVENT2, HistogramVisualizer.HOUR_OF_DAY_BINS)
  visualizer.visualize(stream)
}

For the example log file, which unfortunately has an even distribution of events, we get this:

The cool thing about gnuplot is, that you can actually run these things in a cron job to produce daily reports (and mail them to the appropriate people) or on a continuous integration server to visualise how the system is being exercised by the test suite.

Marc McNeill: SOA, architecture without foundation

Mon, 29 Jun 2009 14:40:53 +0000

Service Orientated Architecture (SOA) is something that is easy for the lay person to understand. (Try getting a techie to explain REST and you will see the attraction of SOA to the business person - it’s understandable!) Understandable, in my non-techie hands, is dangerous. I am entirely unqualified to pass judgment on it, but there are a couple of things I’ve observed and have been on my mind when I’ve seen SOA nastiness going on. So excuse me whilst I wax lyrical.

Problem 1. IT haven’t got a clue.

My (lay) understanding of SOA: IT build ’services’ that can be consumed by different applications. SOA enables us to remove duplication and build the foundations for a scalable architecture that will accommodate changing requirements as the business evolves and grows. Herein lies the problem; IT build ’services’, second guessing what the business actually needs from said service.

Exhibit One: Customer Details Service. It exposes details of customers to any application that will use information about customers. It was designed by the architect in isolation based upon what IT believe a Customer Details Service will be required to do (e.g. the nature of the fields, the domain etc). This is even though no application has been built, yet alone specified for (”we just know we are going to need customer details”). It’s putting the cart before the horse. But IT go ahead and build the service anyway, because they own SOA. At a later date the business articulate requirements for a new downstream application that requires Customer Details. It’s the Corporate Business whose domain is Corporate Customers. But what happens? The service doesn’t quite meet their requirements. The fields are wrong. The Customer Details Service fits the domestic consumer model but not the corporate customers model. What gives? More likely than not the downstream application. The corporate customer has to be shoe-horned into the domestic customer service. I’ve seen this done.

Lesson 1. Don’t build SOA in a void. Get out of that architectural ivory tower and engage with the business (if you can get them to listen - see next point). Better still engage in Guerrilla SOA.

Problem 2. The business haven’t got a clue.

One of the sad realities of the corporate world is that walls that have sprung up and created internal silos that are difficult to bridge. As the business, the consumer of technology, I want IT to deliver to my requirements, no more, no less. If I am in the domestic consumer part of the business, frankly I don’t care about Corporate customers. I’m fighting for my budget, and hell, if this SOA thing is going to cost more than doing a closed application that fits only domestic customers, that only I can use I don’t care. I’m not going to pay for a “Customer Details” service that does anything except give me what I need to know about my customer.

Lesson 2. The architects should facilitate the discussion. SOA is as much about your business vision as it is technical architecture. Unless the business grasps what you are trying to do, drives the solution and requirements are both local and global, before long you’ll see some grand services that few use in the core and chaos is the periphery where the real business is done.

Bottom line? All too often architects fail because they tend to focus upon the architecture part of SOA rather than the services. Unfortunately, because of the siloed nature of so may organisaitons, unless it is driven by the architects it is unlikely to gain traction. If there is a maxim that should be followed when considering SOA in an organisation, it is probably instilling the notion of ‘think local, act global’.

Steven List: Facilitation Antipattern: Repetitor

Mon, 29 Jun 2009 11:11:03 +0000

Motto: It’s worth repeating. It’s worth repeating. It’s worth repeating.

Belief: You’ll only understand if I say it at least three times.

Behavior: Says the same thing repeatedly, frequently in somewhat different words, frequently two, three, or more times.

Characteristics: Articulate, filled with conviction, perhaps lacking confidence

In my last post, this would have sounded like this:

It’s about the subtleties. You know – it’s about the little things. It’s about the stuff that’s not so obvious – the subtleties… the things that others hear in what you say whether you were aware of it or not…

Repetitors are usually articulate. They are able to express themselves. In the positive way, without the unneeded repetitions, a Repetitor would be an Articulate. By repeating themselves, without checking to see whether the listener is understanding, the Repetitor turns a Pattern into an Antipattern.

Dealing with a Repetitor is as simple as a variant on the Facilitation Four-Step: Interrupt, Ask, Redirect, Commit.

Interrupt

“Excuse me, Frank.”

Ask

“Do you mind…”

Redirect

“…if I check in with the others for a moment?”

Commit

“We’ll get right back to what you were saying.”

Action (yes, a 5th step )

“Sue, just so we’re clear, can you tell us what Frank’s point was?”

In this way, I validate that others have heard Frank, check to make sure that they’ve understood Frank, and break the pattern of repetition.

Related Pattern: Articulate

Tweet This Post

Michael Patricios: Rise of the crapplication

noreply@blogger.com (mpatric) — Mon, 29 Jun 2009 10:40:21 +0000

There has been chatter of late in the iPhone developer forums that I frequent regarding the prolifation of crap applications (crapplications, or simply crapps) in the App Store and that the App Store model encourages this. Could this really be true?

Like many others, I have a full-time job and just develop iPhone applications in my spare time. But that does not mean it costs nothing to develop my applications. On the contrary - my spare time is very valuable to me - but how do I quantify it? If you assume that I did some other work, say contract work, over that time then the opportunity cost of this lost income can be determined.

With this in mind, I decided to examine one of my applications, Magnetic Block Puzzle to see what the cost of developing it was and what sort of sales I would need to achieve to break even in a reasonable period of time.

Magnetic Block Puzzle deconstructed

I estimate that it took around 200 - 300 hours to develop, from start to release-ready code. In order to calculate an opportunity cost for this time, I need to determine an hourly rate. Conservatively I am using a rate of £30 per hour. That equates to an opportunity cost of £6,000 to £9,000. For the rest of this exercise, I will use the average of £7,500 (about $11,500). Expenses incurred in developing the application have been ignored.

So, how many sales are required to break even?

.nobrtable br { display: none; font-size: 0.6em; }

Tier	Price ($)	Price (£)	Pre-tax profit (£)	Req sales
Tier 1	0.99	0.59	0.36	20,833
Tier 2	1.99	1.19	0.72	10,417
Tier 3	2.99	1.79	1.09	6,881
Tier 4	3.99	2.39	1.45	5,172
Tier 5	4.99	2.99	1.82	4,121
Tier 6	5.99	3.49	2.12	3,538
Tier 7	6.99	3.99	2.43	3,086
Tier 8	7.99	4.99	3.04	2,467
Tier 9	8.99	5.49	3.34	2,246
Tier 10	9.99	5.99	3.65	2,055

I need to sell 10,417 copies at its current price of £1.19 ($1.99) just to break even! How many sales are required then, per day to break even in six months?

Tier	Price ($)	Price (£)	Pre-tax profit (£)	Req sales per day
Tier 1	0.99	0.59	0.36	114
Tier 2	1.99	1.19	0.72	57
Tier 3	2.99	1.79	1.09	38
Tier 4	3.99	2.39	1.45	28
Tier 5	4.99	2.99	1.82	23
Tier 6	5.99	3.49	2.12	19
Tier 7	6.99	3.99	2.43	17
Tier 8	7.99	4.99	3.04	13
Tier 9	8.99	5.49	3.34	12
Tier 10	9.99	5.99	3.65	11

57 copies a day at the current price doesn't sound too bad until you consider that it needs to maintain this level for six months and that on its best sales day it hasn't come close to this amount. It's looking unlikely that I will break even on this any time soon, if ever.

What are the options then? Raising the price requires fewer sales to break even, but raising the price will result in a decrease in sales too, so this is unlikely to help.

Perhaps lowering the price to 59p ($0.99) will help boost sales? It probably will, but at that price the required number of sales to break even shoots up to 20,833 or 114 a day for six months.

The only other option is to lower the development cost. In other words, spend less time developing it in the first place. Too late for this application, but valuable information to take to the next one. Knocking off 10% or 20% is not going to change the situation much. The decrease needs to be significant (say, an order of magnitude). Putting 20 - 30 hours of effort into building an application means an opportunity cost of around £750, which can more realistically be recouped through App Store sales in an acceptable period of time. But what can be built in 20 - 30 hours? Ah, a crapplication.

Pricing

The above logic of course assumes that a crapplication will sell as well as something that took ten times longer to develop. Unfortunately, the history of the App Store leads me to believe that you are no more certain to sell something good than you are to sell something crap. If it appeals to the masses, it will probably sell, irrespective of how bad or how good it might be (this is the best example of the former).

A popular story that is quoted is the developer that decided to see what could be built in an hour; came up with Sound Grenade in 20 minutes and landed up cracking the App Store top 10 with it.

The blessed / dreaded charts

To break even many sales are required at a low price, or fewer sales at a higher price. In pre-App Store days people would happily pay £5 to £10 ($7 to $15), or even more for a mobile application. Ringtones and images were confined to the price brackets below this. The App Store has changed everything. Customers now expect to be able to buy just about anything for 59p ($0.99). It wasn't always like this - I remember the first few apps on the App Store being more costly. Didn't I purchase Super Monkey Ball and Enigmo for £5.99 ($9.99) each? Yes, I did. What happened?

This slide can arguably be attributed to the App Store charts. The charts are just based on downloads, irrespective of cost. So, a £5.99 ($10.00) application would need 10 times the turnover in order to compete with a 59p ($0.99) application in the charts. Of course, this is a losing battle. Surely charts should take the price into consideration? As it stands, it has clearly become a volume game.

Getting a spot in the App Store charts is essential for every developer. The charts tend to be self-perpetuating. Once an application is in there, it is noticed by more people and therefore downloaded more, boosting its position in the charts, leading to even more people noticing it and so on.

It seems then that developers have little choice but to charge 59p ($0.99) or £1.19 ($1.99), or at a push £1.79 ($2.99) and hope they crack the charts. If they don't, the application is quickly confined to the abyss of unnoticed applications (AUA), picking up the odd few sales here and there, but ultimately not very many.

Summary

It would seem that the best return on investment is to spend as little time as possible building an app, try to give it mass appeal and price it at tier 1 (59p or $0.99). Perhaps then building iPhone software with the aim of making money is viable.

If like me you have spent a significant amount of time building applications, for very little reward, console yourself with the fact that many of us are in the same boat. Let the enjoyment of building the application be your reward and the sales (if there are any) be a nice bonus.

Fabio Pereira: Principle of Relative Priority

Mon, 29 Jun 2009 09:04:03 +0000

Prioritization is crucial for a project to succeed. We need to make sure that we deliver what’s most important, what adds more value. The interesting and most challenging part of this is when we have to ask the business (client, product owner) to prioritize items. By calling them items I mean they are not necessarily stories, features or a product backlog. Sometimes we want to prioritize needs, purpose, goals and outcomes as well. The way we ask them is the trick…

I’ve seen many different ways of defining the priorities and usually there are ranges involved, sometimes with labels or numbers, like:

Very High / High / Medium / Low
Must Have / Should Have / Could Have
Essential / Important / Not very important
1 - 5

There is nothing wrong with creating the labels, but when we ask the priority of one item only, that’s what happens sometimes:

How do we understand business priorities

The most effective prioritization exercise I’ve seen was to give the business a randomized list of items we want to be prioritized, then we ask them to give us the X most important items. And these will be the ones considered during the next timeframe. We should always let the list visible to the stakeholder who is prioritizing… It has to be clear for him that he is giving up on items in favor or others… at least for the next timeframe, which could be the next iteration of release.

It’s like a son’s birthday gift…

Good parents apply this rule with their children. Let’s say it’s close to my son’s birthday, I don’t have a son, this is totally hypothetical :), but my mother used this rule with me… So, I want to buy him a gift, ONLY ONE, of course, because I’m not the kind of dad that gives everything that my son wants. So I have a list of gifts that I know my son wants. Bike and video game are top of the list, but I can give him only one now, because it’s his birthday. We all know It’s close to Christmas (next iteration/release) and I will give him the other one on Christmas, but now it’s his birthday time, only one gift… I, then, make sure that my son understands the one gift per event rule and ask him which one does he want as his birthday gift… As a clever boy, he will think carefully which one really matters more to him because it’s clear that he has to give up on one in favor of the other. He will not give up on it forever though, it’s just until Christmas (next iteration/release), however, he knows that one of them will be bought and received first, and the other will come later…

As I said, there is no problem in defining labels and priority buckets, the only thing we have to bear in mind is that whenever we ask the business to prioritize items, we do not ask them in which bucket he wants to put one single item. And that’s when the principle comes.

Principle of Relative Priority:

All the decisions about item priority have to involve other items which are being traded in favor or the most important one(s). There is no absolute priority, only relative to other items.

One item by itself is usually important, otherwise it would not even be in the wish list… If I ask my son:

Dad: “Do you want a bike?”
Son: YES!
Dad: “Do you want a video game?”
Son: YES!

But when they have to CHOOSE… That’s when the most important items are picked… And during times of crisis, like now, with budgets constraints, we don’t even know if we will be able to afford a Christmas gift, so let’s make our kids happy during their birthday…

Simon Brunning: Links for 2009-06-28 [del.icio.us]

Mon, 29 Jun 2009 07:00:00 +0000

Planet TW

Sam Newman: Currently experiencing the guilty pleasure that is watching The Waterboy. No, it isn’t Punch...

Sam Newman: Given the leading topic, one suspects we may be seeing the end of the trending topics. Crowd...

Sam Newman: I’m officially in love with the shutter sound of the nikon fm3a. Now all I need are two more...

John Hume: AR⊗TA

Felix Leipold: Seaside Event at the London Office this Monday

Mark Needham: Brownfield Application Development in .NET: Book Review

The Book

The Review

What did I learn?

In Summary

Sumeet Moghe: The Agile Elearning Design Manual - Of Project Spaces & Project Project Managers

Growing Versatilists on the team

How environment affects versatility

The waste of documentation inventory

Simplify your work environment

De-managing your project - think "Collective Ownership"

Managing Scheduling

Try a Card Wall

Luca Grulla: What has to be ready for the beginning of a project ?

Ye Zheng: 乐乐呵呵写程序

Manish Chakravarty: Confusion Confusion

Jeff Xiong: AgileChina 2009: Pragmatic Agile

Danilo Sato: Velocity gone wrong #2: Making up points

Jason Yip: 3 imperfect strategies for system test data management

Jason Yip: Some thoughts on roles and hand-offs

Sam Newman: Camera has arrived. eBay, thou art the wind beneath my wings.

Simon Brunning: Links for 2009-07-03 [del.icio.us]

Mark Needham: Domain Driven Design: Conformist

Mark Needham: Coding Dojo #19: Groovy Traveling salesman variation

The Format

What We Learnt

For next time

Jason Yip: Don't just make it visible, make it tangible

Patrick Kua: Be Careful What You Measure

Anand Vishwanath: Challenges in applying Theory Of Constraints to Agile Software Development

Danilo Sato: Velocity gone wrong #1: Done is not done

Nick Drew: nickdrew

Security

Unit Testing

Good news

Puzzling

Continous Integration

Powershell

Asynchrony

Steven List: IAAM: Sympathizing, Empathizing, Identifying

Reaction: Sympathy

Reaction: Empathy

Reaction: Identification

Sympathy

Empathy

Identification

Sympathy

Empathy

Identification

*Reaction vs. Response

Related posts

Sam Newman: I like scala, but binary incompatibility between minor versions? Come on! http://bit.ly/ScalaXBuild

Andy Palmer: Pomodoro, AppleScript and Adium

Mark Needham: F#: Pattern matching with the ':?' operator

Andy Palmer: Tiny Types

Sam Newman: Is #mongodb too good to be true? Where’s the catch? Ah - their website is powered by...

Mark Needham: Book Club: Logging - Release It (Michael Nygaard)

Dan North: The perils of estimation

We’re missing the point of release planning

Redefining success – in a bad way

Back to project basics

Use the inception for deliberate discovery

Pankaj Nakhat: My Childhood

Sam Newman: Just watched: #Yellowstone on BBC4. Simply stunning - rivals Planet Earth for beauty. No idea why...

Martin Fowler: RequestStreamMap

Sam Newman: Just won on ebay: Nikon FM3a. Might have to freeze myself and get a friend to thaw me out for its...

Sam Newman: Just read: #D-Day by Antony Beevor. By turns harrowing, entertaining and insightful. Recommended.

Sam Newman: Now reading: Agile Estimation and Planning by Mike Cohn. Off to a dry start - I hope it picks...

John Hume: A Quick Look at Azure, Microsoft's Cloud

John Hume: Clojure on Google AppEngine

Creating Your Application

Hello, World!

The User Service

Datastore