Venice Photos Coming Online

Andrew Johnston — Sat, 20 Jun 2009 06:44:08 GMT

I've just published some of my photos from my trip to the Venice Carnevale 2009. There are some more scenery and miscellaneous shots to follow, but if you like pictures of pretty ladies in striking costumes, look here.

Legislative Stupidity

Andrew Johnston — Tue, 07 Apr 2009 20:08:16 GMT

I feel compelled to write this post to highlight a stupid law which rather than constraining terrorists hands them victory.

In July 2008 my wife and I attended the Northampton Balloon Festival. An enjoyable time was had by all, notwithstanding a complete absence of balloons, occasioned by the British weather.

One of the highlights was a successful attempt on the world record for the greatest number of performers singing "Y.M.C.A"! This shot captures the scene:

OK, it's not "Afghan Girl" or "Migrant Mother", but I liked it, capturing a British Bobby policing his community exactly as required.

This photograph is legal. Had I taken it after February 16th 2009 I would have committed a crime. The stupidity of that law hands victory to the terrorists.

Andrew

Website Returns, and New Photos

Andrew Johnston — Fri, 06 Mar 2009 07:44:46 GMT

Apologies for the state of the website over the last week - a problem with the hosting server. I seem to have worked around it for now, so thanks for your patience and those who alerted me to the problem.

My photography zone has been updated with images from my trip to Maine, Vermont and New York last autumn, and from the dramatic Winter weather in the UK. Photos from my trip to the Venice carnivale should start to follow shortly.

Quantum of Disappointment

Andrew Johnston — Sat, 08 Nov 2008 07:11:15 GMT

I don't often review films, but I think someone has to cut through the sycophantic hype and say it: Quantum of Solace is c**p.

This isn't a Bond film, it's like a bad entry in the Bourne series. Where is the elegance, the charm we should expect of Bond? Bond films have always traditionally leavened the action with humour and beauty. Both were spectacularly missing from this episode. Read the rest of my review...

Underwater White Balance

Andrew Johnston — Fri, 25 Jul 2008 08:41:35 GMT

If you're lucky enough to own either a waterproof camera, or a waterproof housing for your digital camera, you'll have experienced the problem of trying to correct for the extreme blue cast of shots taken underwater.

The problem is that there's little reliable advice on how to correct this. There are a few articles on the web, but I haven't found any to provide a reliable and repeatable process which can be applied to "casual" underwater photography.

After a fair amount of research and experimentation, here's my analysis of the options available, and a suggested technique which I have developed and which I haven't seen documented anywhere else.

Photography Blog

Andrew Johnston — Fri, 25 Jul 2008 07:40:20 GMT

I've now created a separate category for photography-related blog posts, so if you're interested you can track these separately

A Shortage of Analysts?

Andrew Johnston — Wed, 11 Jun 2008 19:11:28 GMT

I've just spent two days at the 2008 Enterprise Architecture Conference in London. It was a very high quality event, with a range of speakers covering topics from pragmatic analysis techniques to how to manage knowledge through the life of NASA's Mars programme, more than any single working lifetime.

Overall there was much less focus on technology (read SOA and modelling tools) this year, and a vigorous and renewed focus on business alignment and business architecture, which, if we can deliver, potentially places architecture where it should be, as the business's agent.

But there's a problem. Good business analysis is fundamental to this, yet several delegates bemoaned the current lack of good business analysts. User organisations often struggle to articulate and abstract their needs, and this feeds into all downstream processes. Modelled requirements are an increasing rarity, poorly substituted by imprecise verbal statements in Word or PowerPoint.

The problem is, of course, not unique to analysts, and may have common cause with the equal lack of architects. Senior architects and analysts both tend to have several big birthdays under the belt, and many learned their trade as developers, gaining both practical method skills and the experience of turning ideas into working code. (The majority of exceptions have other "making it work" experience, such as building networks or running data centres.)

But in the current world of ERP packages and large-scale outsourcing, many organisations no longer build anything themselves. The live classroom has been thrown away.

I have worked with a number of good, keen young analysts, but most work for large supplier companies who still have both well-funded training programmes and the breadth of work to build experience and a broad skill set. These guys and girls can do a good job, but at the risk of higher costs and potential conflicts of interest.

We already know that this may reduce organisations' ability to ensure the right solution to their needs, or assure its quality. Recent observations suggest that organisations who forgoe getting their hands dirty in IT will also suffer an increasing difficulty in creating a clear, concise and structured statement of those needs themselves.

My Grand Circle Tour

Andrew Johnston — Wed, 02 Jan 2008 14:51:01 GMT

I've just added a new section to my photo gallery, and an article describing my recent "Grand Circle" tour of the Southwestern USA: what worked, what didn't, and how to avoid sleep deprivation and scurvy!

Using Volume Shadowing with Ntbackup Under Vista

Andrew Johnston — Mon, 09 Jul 2007 00:00:03 GMT

The brain-dead backup function of Windows Vista is enormously annoying. There are known ways to get good old ntbackup working, but they have their limitations. Read this article about my attempts to get round some of those limitations.

My Photo Gallery is Now Live

Andrew Johnston — Wed, 04 Jul 2007 06:19:21 GMT

I've always had a strong interest in photography, both as a practical tool to record our experiences in life, and as an art form. Fans of the Coppertrees Calendar will know how I've had great fun capturing friends, family and travel over the years.

I've also always had great admiration for those photographers whose work can really move the emotions, particularly the great landscape photographers like Ansel Adams. Taking good representational photographs is hard enough, but establishing a composition which can both emotionally inspire the viewer and satisfy the critics is even harder.

Over the last year or so I've been rediscovering photography with a Digital SLR, and on my new photography pages you'll find some of my efforts, plus (in coming weeks) some hints, tips and tools for the D-SLR photographer.

Paradigm Shift - Clear Memory Now!

Andrew Johnston — Sat, 02 Jun 2007 01:01:00 GMT

I've been musing lately on why we in IT insist on forgetting so much valuable knowledge. I don't know whether it's because of our youth-obsessed culture and our focus on the newest and best, because of our tendency to prioritise on-the-job over traditional learning, or whether there's simply too much in the "architect's book of knowledge" (ABOK), and we all have to focus on the new to keep up.

I explore two very different examples: the value of understanding RS232 in this 3G+ world, and some recent discussions on service reliability, both of which can be resolved using some quite old knowledge.... (Read More...)

New Book reviews

Andrew Johnston — Fri, 27 Apr 2007 08:59:40 GMT

I've just added four book reviews:

"Freakonomics" by Steven D Levitt & Stephen J Dubner
"Examples: The Making of Forty Photographs" by Ansel Adams
"Dark Matter" by Philip Kerr
"Sevent Ancient Wonders" by Matthew Reilly

The Tevye Scale of Approval

Andrew Johnston — Fri, 27 Apr 2007 07:36:25 GMT

The accept/reject assessments of the Sarbanes-Oxley world are far too binary, as they don't allow an architect to record his true feelings about a piece of work. I have therefore decided that in future I will record my assessments using what I have named the "Tevye Scale of Approval"

Annoyance-Based Technology Selection

Andrew Johnston — Mon, 16 Oct 2006 17:36:32 GMT

I'm becoming increasingly frustrated with some day to day technology, and I'm not alone. When you live with technology day in, day out, what it does well becomes taken for granted. What it does badly becomes its defining features. If the annoyances are too great or numerous you will seek a replacement, or even give up entirely. This article explores my own tales of woe with that most commonplace of technology, the mobile phone...

Enterprise Architecture Conference 2006 - My Paper

Andrew Johnston — Tue, 20 Jun 2006 08:10:29 GMT

I've just spent three enjoyable days at the 2006 Enterprise Architecture Conference in London. IRM did their usual excellent job of making it run like clockwork, and my good friend Sally Bean helped them develop an interesting and varied programme. To my mind the best speakers were Jeff Scott, and Chris Wilson of BP. Another encouraging sign was the presence of a great many International delegates.

I presented a paper on Agile Architecture. If you regularly read my work you'll recognise many of the ideas, but I've managed to bring them all together for the first time. You can download my slides and script here.

What was very interesting was how the thrust of the material has changed from a few years ago. No-one was claiming that a given framework, process or toolset can solve EA problems. At the risk of being uncharitable I thought John Zachman's ideas sounded very tired, and there was almost no mention of alternative frameworks such as TOGAF. I may have self-selected by not attending any vendor sessions, but there was also no promotion of tools or technology. A common view was that EA, SOA and many supporting concepts are currently entering the trough of the hype cycle.

Instead the focus was largely on people-related problems and approaches. The labels varied, but several speakers introduced ideas familiar to agile architects. Maybe we're doing something right after all.

You Need Architects...

Andrew Johnston — Wed, 24 May 2006 18:24:41 GMT

Just in case you haven't already seen it.... Why you need architects, in song and dance. Enjoy!

Best Practices in Test Automation

Andrew Johnston — Tue, 23 May 2006 16:12:38 GMT

I am looking for one of my clients into how costs can be reduced, or quality increased, by increasing the extent to which testing is automated.

As a first step, I am trying to develop a comprehensive list of test automation "best practices", grouped roughly by life-cycle (or iteration) stage. I'm trying to find practices which are broadly independent of specific methods and technologies, although obviously tool support may vary depending on the chosen technology.

The following is my first draft of such a list. The tools mentioned are simply examples - there are several tools available in each category, both full cost and open source.

I'd welcome suggestions from my readers if you think there are any omissions (or if you substantially disagree with anything I've included).

Thanks

Andrew

Unit and System Testing

Suite of automated unit and system tests, which can be run on demand by the developers and before integration.
Test-driven development process, in which unit tests are developed before the code they test.
Unit and system tests built into the product suite, using one of the xUnit family of tools.
Daily or continuous integration process, which includes execution of the test suite.
Use of virtual machines or other "cloning" technologies to create easily reproducible development and test environments.
Code coverage measured using a tool such as NCover, and demonstrated to be 100% (or as near as practically possible) for all code.
Code coverage validated using mutation testing (using a tool such as Jester) to confirm that every line can affect the outcome of at least one test.
Feedback process to enhance test suite for bugs discovered in later test stages or live use.

Integration Test

Automated end-to-end tests, run for every release.
Functional coverage measurement, possibly combined with traceability mechanism between requirements and tests.
Ability to easily create and reset an environment which functionally matches production.
Automated test management and reporting, providing control/confirmation of tests run, pass/fail results, errors and diagnostics and resolution tracking.
Repeatable and automated performance and/or stress tests
Ability to inject/extract test messages and files into and out of interfaces and to track them through any integration scheme, ideally in production as well as test environments.
Repeatable, pre-prepared set of test messages/files which test all interfaces, including exception handling.
Feedback process to enhance test suite for bugs discovered in later test stages or live use.

UAT

The nature of user acceptance test is that most functional testing will be manual, since this will provide maximum independent assurance to the users. The following focuses on where automation can best support the process:

Ability to easily create and reset a UAT environment which functionally matches production.
Automation for set-up, background and repetitive tests.
Automated test management and reporting, providing confirmation of tests run, pass/fail results, errors and diagnostics and resolution tracking.
Ability to inject/extract test messages and files into and out of interfaces and to track them through any integration scheme, ideally in production as well as test environments.
Feedback process to enhance test suite for bugs discovered in live use.

Production Support

Automated and non-destructive end-to-end and integration tests, which can be run on demand following any production change, or as an initial diagnostic action.
Ability to inject/extract test messages and files into and out of interfaces and to track them through any integration scheme.
Regular automated execution of key tests to verify system performance.
Automated test management and reporting, providing confirmation of tests run, pass/fail results, errors and diagnostics and resolution tracking.

The Agile Architect at EAC 2006

Andrew Johnston — Wed, 22 Mar 2006 20:00:49 GMT

If anyone is interested in hearing more about my views on architecture, and how agile methods apply to the work of the architect, please sign up for the 2006 Enterprise Architecture Conference in London in June.

I'm presenting a paper entitled "The Agile Architect". This focuses on both how agile projects can have a strong architecture, and how architects can learn and benefit from agile approaches. I take a rather different approach to some recent papers with a similar title (e.g. at this week's otherwise excellent Microsoft Architecture Insight conference), which suggest that agile projects can "do away with the architect".

I look forwards to seeing you there.

Andrew

An Agile Architecture War Story

Andrew Johnston — Tue, 03 Jan 2006 00:00:00 GMT

I don't really believe in a common architectural process. As the author of a successful project management book, and recent articles on data architecture methods, I probably shouldn't say this, but to paraphrase a famous quote, "When I hear 'process', I reach for my gun!"

This is a story of a project I worked upon which followed an informal, agile process, but delivered a successful architecture. Hopefully it serves to support my assertion that agile can have an architecture, but needs an agile architect.

Who Are the Architects?

Andrew Johnston — Thu, 19 Jan 2006 00:00:00 GMT

There's a perennial discussion in architecture forums like the WWISA about the role of the architect, and the discussion regularly degenerates into a debate between the broad and narrow views of what the architect does.

But I'm not sure that's the key question. I think the right question is "Who are the architects?"

Somehow, a number of tasks must be discharged, but how varies from project to project. In the last year I've had a modest building project which tells an interesting story about how different people contribute to "the architecture". Read more here...

Happy New Year!

Andrew Johnston — Tue, 03 Jan 2006 07:35:49 GMT

Apologies to regular readers who wondered if I'd fallen off the planet. It's been a busy second half of 2005, and I haven't managed to do much reading or writing for a few months. However, with the builders gone and a very difficult year over, hopefully things will get back to more nearly normal!

Sorry, there's no Coppertrees Calendar for 2006 - another casualty of our very busy 2005. Hopefully it will make a return in 2007.

I'm currently working on a new paper, looking at the different ways of modelling dependencies in software. I'm hoping it's possible to merge some of the matrix-based techniques with pattern-based approaches - watch this space for more details.

A very Happy New Year to all my readers. Here's hoping that we'll all have a slightly easier 2006.

Modelling Data Mapping - A Challenge

Andrew Johnston — Fri, 26 Aug 2005 15:44:46 GMT

Almost all integration projects contain one or more transformations (sometimes called "mappings") between two different structures holding equivalent data (for example the order tables in the database, and the order XML message). We know how to model the individual static data structures in various ways, but the most common approach is to represent each by a UML class model, and there are established conventions for how to do this for different data sources.

However, UML doesn't help when it comes to the transformations themselves, and typically the detail has to be captured either in code, or a proprietary format. Most good integration tools provide some sort of "visual mapping tool", where the developer drags and drops to create links between representations of the two structures, usually imported directly from their physical schemas. Here's an example using SeeBeyond. Altova provide a good stand-alone data mapping tool called Mapforce - here's an example showing it in use. The problem is that these tools work directly with the physical structure, and don't export the mapping information in a reusable format, so that information is completely disconnected from the UML analysis or design models.

I have experimented with trying to represent mapping information in a UML model, but so far without much success. The best solution I've found so far is to use some sort of "pseudo code" (it could be OCL, pseudo-Java, pseudo-VB or anything similar). For example, we could easily annotate the model with code fragments such as:

Database.order_table.order_no = Message.Header.OrderNo

(where each element refers to a UML Package.Class.Attribute combination).

The problems are that it's not clear where to put this annotation, most UML modelling tools won't help generate it, and there's no graphical representation. Ultimately, writing pseudo-code like this is probably not much better than abandoning the model and moving straight to using your integration tool.

My question is: does anybody know a better way? Has anybody found a good way of representing mapping information in UML? And if so, is there any good tool support?

If you know, please send me a message.

Review - A Short History of Nearly Everything

Andrew Johnston — Tue, 16 Aug 2005 08:03:50 GMT

I've just posted my review of Bill Bryson's "A Short History of Nearly Everthing". I found it an excellent holiday read, athough a general science book with almost no illustrations or equations took a bit of getting used to. For more, please read my full review.

Metropolis - Where Do You Want To Live Today?

Andrew Johnston — Mon, 08 Aug 2005 15:57:02 GMT

There's been a lot of talk in recent years about a "city planning" metaphor for Enterprise Architecture development. Pat Helland's article "Metropolis" in the Microsoft Architecture Journal is a very good example (see my post on this for some key quotes).

While the metaphor might still be valid, some people are beginning to question how far it should be taken. Helland's article, like others before it, implies that "good" EA looks rather like a medium-sized modern American town, complete with relatively standard services, civic buildings and commercial venues. In an answer to the original "Metropolis" article Richard Veryard and Philip Boxer have published "Metropolis and SOA Governance" which challenges several of Helland's assumptions.

I think that maybe we should extend the metaphor by thinking about cities, or Enterprise Architectures, as very diverse entities. What sort of "city" do you live in? To what extent is it planned? What is the vision, and do the citizens share in it? Does the EA resemble a nice neat midwest town, a dark, brooding Gotham City, a glass and steel Utopia, a federation of small towns with lots of empty space between them, a medieaval walled town, or a wartime mid-european ghetto?

And the metaphor can be taken further. Do you want to promote "infill development", closing up functional gaps, or do you want to keep clear separation between the various zones? Do you want the shared services to be clearly visible, as they are in modern, purpose-built towns or hidden beneath a facade which looks much older or simpler? Do you expect to eventually knock down and rebuild older "legacy" zones, or do you want to preserve them with the minimum of change (a common requirement for our valuable historic buildings)? Do you want to accomodate the small hardware shop (read small the bespoke system) as well as the new DIY superstore (the ERP package)?

Finally, remember that it is extremely rare for a city to be truly planned and designed from scratch. You usually start with something established. Even if the city has been flattened by a bomb, you'll have to observe land rights (this is what stopped Christopher Wren and Charles II realising their grand design after the Fire of London). This is equally true of Enterprise Architectures.

The city planning metaphor is a powerful one, but its true power may come if we use it to explore problems as well as utopian ideals.

My Favourite Films - An Exercise in Over-Analysis

Andrew Johnston — Mon, 08 Aug 2005 00:00:00 GMT

A bit of a change from my more serious posts, but maybe a useful lesson in analysis, here's the sorry tale of just how complicated I managed to make listing my top ten favourtite films. I hope it gives you a giggle.

Review - Enterprise Integration Patterns

Andrew Johnston — Sun, 24 Jul 2005 08:45:53 GMT

I've just posted my review of Gregor Hohpe and Bobby Woolfe's excellent book on Enterprise Integration using messaging, "Enterprise Integration Patterns". Overall it's an excellent book, and wiil probably become a "bible" for those involved in the high-level design of integration solutions. To find out more, please read my review.

Change of Links - An Apology

Andrew Johnston — Tue, 19 Jul 2005 18:13:23 GMT

Apologies to my regular readers if it looks like a lot of items have changed. I've just created "PermaLinks" for each entry in my blog, so you can read my wisdom in the future even if I re-organise my thoughts page!

Keep reading!

Andrew

Interfaces and Document IDs - A Rant

Andrew Johnston — Mon, 18 Jul 2005 09:08:30 GMT

Please forgive me if this sounds like a rant, but I'm very annoyed. Someone who should know better has without warning changed a public interface, with the inevitable effect that dependent systems, in particular my blog, have broken. The offender? The mighty Microsoft.

Regular readers will know that I've highlighted several articles published in Microsoft's Architecture Journal. A week ago I went to post a note on another article, to find that all my previous hyperlinks were broken. Thanks to my regular correspondents Richard Veryard and Arnon Rotem-Gal-Oz I discovered that the cause is an internal reorganisation within Microsoft, and there is a new web location for the journal, although it wouldn't surprise me if that changes again. (To add insult to injury, the new URLs are very cryptic, and don't paste easily into my blog!)

Now if you follow Microsoft's advice when building systems, interfaces should be immutable. Otherwise you just don't know what will break. The Microsoft advice is to never change an existing interface - if you need a different one, create a new interface, or at least a different version. And maintain the old one as long as dependent systems need it.

Microsoft are actually very good (not perfect, but quite good) at following this rule in their software systems. But they don't seem to understand that the same rule should apply to that big public interface called the website. There are, of course, perfectly good strategies which would avoid this problem.

First, don't try to reflect internal structural changes in the MSDN website. Doing so is like changing a system's interface just because the implementation has been updated - the opposite of good practice. The public interface should be independent of implementation details.

Second, if you must create a new interface, keep the old one working. In systems, you can usually wrap the new interface to mimic the old. The same is true for a website. A set of auto-redirect pages at the old addresses, and I would never have even noticed the change.

Unfortunately Microsoft have done neither of these. And they seem to have a corporate blindness to the fact that documents are interfaces too. MS SharePoint is based on a web idiom, in which documents are identified by their position in a hierarchy. Re-arrange the hierarchy, and any external references or cross-references suddenly break.

Professional-quality document management systems don't do this. They identify and control documents via a unique, immutable key into the underlying repository, and the primary document access via this key is guaranteed. Of course, you also want to show the document in a hierarchical structure, but any such entry is just a pointer to the underlying document. And if you want to change the hierarchy, or expose the same underlying document at multiple places in multiple hierarchies, it's easy to do. The world of blogging has a similar concept, with "permalinks" which (should) survive a reorganisation.

Memo to Microsoft: links are interfaces too!

Metropolis - a Metaphor for IT Maturity

Andrew Johnston — Mon, 11 Jul 2005 17:25:16 GMT

I've just read an excellent paper by Pat Helland of Microsoft, in which he likens the development of cities and manufacturing in the 19th century to the development of systems and business models now. His conclusion - IT at the moment is about at the same stage as America in the 1880s, when they were just starting to turn the Wild West into an industrialised nation!

Three short quotes from Helland's conclusions bear repeating directly. On heterogeneity he says:

Remember that heterogeneity happens. Unless you have a very simple application portfolio, shared services will not be achieved by trying to put all of your applications on one version of one platform. Even if you could, the next merger would change that! Rather, you have to design for interoperability and integration across platforms. This is the force that is driving the industry wide work in service-oriented architectures.

He extends the popular "city planning" metaphor to IT investment:

IT investment is a balance of funding the sacred, protecting historic monuments, and allocating spending between infrastructure and business opportunity. Striking this balance is a key facet in effective governance, and in realizing the potential of IT in your organization.

And finally, those who seek to maintain control of their enterprise architecture through heavy governance would be well advised to note:

You have to maintain a light hand. It is counterproductive to try to dictate what happens in every structure in town, what color shirts are made, and how much is charged for soap. You have to embrace the semi-autonomous approach to governance that is characteristic of our cities, and allow the process owners to optimize and achieve efficiencies with as few constraints as possible.

Death of the Microsoft Architecture Journal?

Andrew Johnston — Mon, 11 Jul 2005 17:18:45 GMT

Does anybody know if Microsoft have killed off their Architecture Journal? I was just about to write a post linking to it, and I find the content has been moved to an archive area and all the links have changed. Please send me a comment if you know!

Update

Thanks to my regular corrspondents Richard Veryard and Arnon Rotem-Gal-Oz I've managed to track down what's happened. The journal is now at http://www.microsoft.com/architecture/default.aspx?pid=journal

This "broken interface" breaks so many architecture rules it deserves a separate post, so I've written one!

Cirrus Minor - A New Architecture Site

Andrew Johnston — Tue, 28 Jun 2005 07:43:23 GMT

Arnon Rotem-Gal-Oz has set up an interesting new site / blog dedicated to software architecture. Of particular note, he's trying to put some detail on the architecture "process" which is often negelcted as a single box on the development process picture. His approach has the name SPAMMED, catchy, but might cause the odd problem with email filters :)

Domain-Specific Languages

Andrew Johnston — Sun, 26 Jun 2005 09:06:19 GMT

There seems to be quite a lot of activity on the "Domain Specific Language" front at the moment. Martin Fowler published "Language Workbenches: The Killer-App for Domain Specific Languages?", in which he concludes that the common programming pattern of setting up repeating data structures via either very similar lines of code, or an external configuration file, is actually a DSL. He also republished a paper by Dave Thomas entitled "Design to Accomodate Change" on the related topic of table-driven programming.

However, Martin's essay goes beyond common programming and data techniques to look at the development of specialist tools which he calls "Language Workbenches". I'm not completely convinced that we need these in the world of XML and XSD. If you have a defined schema for you XML-based DSL (and aren't all the many *ML langauges just different DLSs?) then any schema-sensitive editor will provide you with good design and editing support. The leading IDEs (e.g. Visual Studio) all have such a tool built into their core capabilities. Surely we now have a sufficiently sophisticated set of XML-based tools and standards that we have an opportunity to exploit synergies rather than re-inventing the wheel?

The Fear Premium

Andrew Johnston — Thu, 23 Jun 2005 18:56:51 GMT

In an interesting echo of my last piece (Why Software Isn't Like Building Construction), Scott Ambler has analysed bureaucratic processes as a response to management fear about what can go wrong in software development. His conclusion is that these processes only give the illusion of addressing the underlying fear. His article is well worth reading.

Why Software Isn't Like Building Construction

Andrew Johnston — Mon, 20 Jun 2005 00:00:00 GMT

Many software development and management methods are founded on a basic assumption - that constructing software is rather like building a bridge or a house. Once we've "done the design", actually generating the software ought to be a completely predictable, relatively low-skilled process. However four decades of failure to achieve this vision might suggest that we should revisit the assumption.

In a paper entitled "The New Methodology" Martin Fowler, the guru of object-oriented development, suggests a couple of reasons why this might be.

My article answers Martin's, suggesting a couple of other considerations, and whether we have to completely abandon the physical construction analogy as a result.

Application Development Strategies

Andrew Johnston — Sat, 21 May 2005 00:00:00 GMT

I recently attended a day of the Butler Group "Application Development Strategies" Symposium. I've just posted a short report on some of the more interesting discussions and presentations.

Almost all of the presentations shared a reminder that we still have a "software crisis" - the vast majority of software projects fail to deliver to their original targets and estimates. The presentations suggested three independent, but not exclusive, approaches to try and resolve the problem:

Adopting better, more agile processes to address fundamental weaknesses in "waterfall" processes,
Adopting better tools and techniques to improve development productivity and the integration of the application life-cycle,
Enforcing a stronger "enterprise architecture" framework for development.

This last one was surprising, with several papers echoing my view that a strong architecture is essential if agile development is to succeed on a large scale or in complex and critical applications.

There was also surprising agreement on things which won't solve the problem:

No-one was promising a technical or product "silver bullet". This includes web services!
No-one was suggesting that we should just "try harder" with old-fashioned tools and processes.
There's no "one size fits all" solution. For example it's a mistake to force a formal, high-ceremony process onto small business systems developments.
Excessive technical standardisation is also not the answer. The drawbacks include "lowest common denominator" technical solutions and inflated costs where the standard solution is "overkill".

Read my report for more details.

Review - My Early Life

Andrew Johnston — Wed, 18 May 2005 19:26:43 GMT

I've just posted my review of this wonderfull book, by one of the world's greatest leaders. The book is exciting, inspiring and, most of all, fun. I urge you to read it (and my review)!

The Laws of Identity

Andrew Johnston — Wed, 18 May 2005 14:49:49 GMT

Microsoft have just published an excellent paper by Kim Cameron discussing the characteristics of an "identity metasystem" which must evolve if we are to have proper trust in the Internet and interactions which take place through it.

The paper is also available from the Identity Blog.

The paper's thrust is that we need to develop a unifying set of identity-related technologies, but that these must observe certain key "laws", and must accomodate varying technologies and requirements, much as unifying APIs provide access to a variety of hardware technologies.

I started thinking about the most common form of digital identify at the moment, the email address. It can be used in accordance with many of the laws. I can (usually) control when I release it. I can have different identities in different contexts, and choose which one to disclose. The identity is verifiable (to a limited extent) - someone can send me mail to check my address is valid. A variety of service providers and technologies are supported.

The big problem with email, of course, is that I can't usually verify that email is from the claimed sender. For example, my spam whitelist admits email apparently from microsoft.com, but some of these emails are offers of dodgy mortgages and promises of increased manhood, obviusly not from the claimed source!

As a result, I wonder whether there is a missing "law of identity". I need to be able to verify a claimed identity by methods I trust. I'd express the law something like "A party must be able to validate any identity claim, particularly its ownership, by reference (directly or indirectly) to resources he or she trusts." This is implied in the current laws, but might be important enough to promote to a law in its own right.

Growing a Language

Andrew Johnston — Wed, 11 May 2005 20:08:56 GMT

I've just read a wonderful paper by Guy L Steele, "Growing a Language". He argues strongly that programming languages must be "small", but able to grow. Such a language will have a relatively simple structure, syntactic rules, and a small core vocabulary. However it must also be able to "grow", integrating new data types and functions, in the form of user code accessed in exactly the same way as core functions.

Steele's argument is highlighted by the paper's unique style - he uses a small but growing language himself. He writes using only words of one syllable, gradually adding other words he has defined in these terms.

The paper works at many levels. As well as the fascinating intellectual exercise in style, it makes a strong case for:

simple but extensible programming languages,
improving the extensibility of Java, rather than its core vocabulary,
an agile community process for developing languages, rather than up-front design of great monoliths,
the communication power of simple language and words.

Steele exhorts us to think about extensibility mechanisms - if we get these right then the core functionality can always grow. And by example, he encourages us to use simple, direct language and benefit from its discipline. On both accounts I agree wholeheartedly.

Valuing Flexibility

Andrew Johnston — Mon, 02 May 2005 17:34:37 GMT

I've just posted my review of "Real Options", a book about financial market techniques to put a quantitative value on the benefits of flexibility. Unfortunately I was rather disappointed by the book - read my review to find out why.

However, reading the book has prompted me to do some more research in this field. I'm currently looking at whether combining the core valuation equation with probability decision trees might be useful. I'll post more when I have some results.

Why Do Buses Come in Threes?

Andrew Johnston — Mon, 28 Mar 2005 08:21:53 GMT

I've just posted my reviews of two excellent books by Rob Eastaway and Jeremy Wyndham. "Why Do Buses Come In Threes?" and "How Long is a Piece of String?"

The messages of these books are that mathematics is important to everyday life, and that it's fun. Have you ever wondered why flowers often have five petals, how bookies' odds work, how you always end up in the slowest queue, or, indeed, why buses come in threes? If so, then this is the book for you.

Whether as an introduction if you've never enjoyed maths before, or a reminder if you have, I thoroughly recommend both books.

Enterprise Integration Patterns

Andrew Johnston — Fri, 25 Mar 2005 00:00:00 GMT

Integration, like other design activities, can benefit from sharing ideas and proven strategies in the form of patterns. An excellent starting point is Gregor Hohpe and Bobby Woolf's Enterprise Integration Patterns website and book.

In my recent work I've discovered a few patterns of my own, and I've started a page to document them.

First up is entitled "Change Indicator" . You may have a legacy system with an EAI adaptor, or a similar source of messages, which reports on the current state of key business objects. The messages will typically tell you when something has changed, but not necessarily what has changed, but the latter may be important to downstream systems. This pattern shows how to use EAI elements to add this information in a way which is totally transparent to the legacy system and existing users of the EAI scheme.

I Told You So

Andrew Johnston — Sun, 20 Mar 2005 11:02:14 GMT

In Extensibility Points, Gregor Hohpe describes architecture as a guessing game, trying to understand future changes to a business, and designing a system to cater for those changes. He's written a good article on types of extensibility and where they can be applied.

But Gergor's article doesn't really touch on understanding the dynamics of business change, and how these should dictate the form of architectural extensibility. Neither does he discuss the problem of being an architect who understands what flexibility is required, but being unable to get either the customers, or the suppliers, to agree. I seem to spend a lot of my time saying "I told you so" when a problem I predicted occurs - this is frustrating, even if I'm being proven right.

If you want to understand more about the dynamics of change, you might look at my previous postings on the subject:

Architects - Masters of Order and Unorder

Business Flexibility

Strategies for Flexibility

Web Service Challenges

Andrew Johnston — Sun, 20 Mar 2005 00:00:00 GMT

In a recent article, Gregor Hohpe asks "Is SOA Like Drunk Driving?" In our attempts to address the shortcomings of component-based development have we "swung too far" and introduced new problems?

One recent experience suggests to me that we may well be at risk of this. Read about my first experience of Web Service development and judge for yourself: have we thrown the baby out with the bath-water?

Blogging: What It Is and How It Works

Andrew Johnston — Thu, 17 Mar 2005 22:06:54 GMT

Another good article published in the Microsoft Architecture Journal : If you don't fully understand what a Weblog (or "Blog") is, how it works, or what it may mean to you and your business, then this is for you.

The first part of "DasBlog: Notes from Building a Distributed .NET Collaboration System" is an accessible review of the phenomenon of blogging, and its implications for collaboration and knowledge management. Blogs cross several traditional boundaries: diaries, content management, collaboration, and news publishing. As a result both authors and readers are changing the way they get news and opinion from the web, and the new patterns are beginning to be reflected in corporate and project communications.

The second part of the article discusses some of the underlying technologies, like RSS, and how they can lead to increasingly rich linkage between the work of many authors.

Finally, the third section discusses the challenges of building a full-function open-source blogging engine using .NET technology. It's very interesting, but gets quite technical. However you don't have to follow, or even read, this to get benefit from the excellent first part.

Enterprise Architecture Design and the Integrated Architecture Framework

Andrew Johnston — Tue, 08 Mar 2005 20:10:06 GMT

I've recently been catching up on the Microsoft Architecture Journal. This is an occasional MS publication, which can be downloaded from the .NET Architecture Center. It's got a lot of good articles, with a software architecture and process focus.

Of particular interest, there's an article in the first edition by Cap Gemini on "Enterprise Architecture Design and the Integrated Architecture Framework". Obviously every consultancy has their own architecture framework, and they all share some features, but what makes this one a bit different is the strong focus on the contextual (business context), conceptual (vision) and logical (solution independent capability) aspects, with technical details relegated to a bottom physical layer. It also has one of the best discussions I've seen on the relationship and differences between enterprise and project architectures.

Blogger Me!

Andrew Johnston — Mon, 07 Mar 2005 20:14:47 GMT

My blog is now fully live. My Thoughts on the World, and the articles in AgileArchitect.org, are now available via an RSS feed.

For the technically inclined, I've built my blog using a combination of VB.NET, Active Server Pages, and a bit of XSLT. This allows me to develop my site using FrontPage and IIS, and run it under Apache. If anyone's interested I'll write a technical note on it.

Domain-Specific Modelling Techniques

Andrew Johnston — Fri, 18 Feb 2005 01:00:00 GMT

Sometimes formalised modelling techniques like UML are not the best solution. Microsoft have recently started talking about "Domain-Specific Languages", specialised modelling techniques for specific tasks. My Domain-Specific Modelling Techniques page outlines some useful DSLs I have developed or discovered.

Business Flexibility

Andrew Johnston — Wed, 23 Feb 2005 02:00:00 GMT

Inspired by an article on Richard Veryard's SOAPBox Blog, this piece considers the business equivalents of expansion capabilities like the spare slots in a desktop PC.

A Fast Diff Algorithm

Andrew Johnston — Mon, 21 Feb 2005 19:00:00 GMT

This recent posting to The Code Project is an implementation of a Diff algorithm in VB.NET, with various techniques to improve performance, while keeping the code simple.

Architects - Masters of Order and Unorder?

Andrew Johnston — Fri, 18 Feb 2005 15:00:00 GMT

Do you work in an ordered environment, where things follow rules in a nice, predictable way? Or does your environment exhibit "unorder", characteristics of complexity or chaotic behaviour. If the latter, you need to read this paper to understand how your domain may differ from others, and appropriate techniques to use as an architect, analyst or designer.

Architecture, Design and Engineering

Andrew Johnston — Wed, 16 Feb 2005 01:00:00 GMT

Is IT architecture about the "hidden bits of plumbing", or should it focus on the "visible design" delivered to users, developers and other stakeholders? Read this paper, and consider your view.

Modelling an Enterprise Data Architecture

Andrew Johnston — Sat, 06 Dec 2003 08:00:20 GMT

Unlike the simplistic models in books and training courses, a real enterprise has a very complicated data architecture. Most of the data will be held in large legacy or package systems, for which the details of data structure may be unknown. Other data will be held in spreadsheets and personal databases (such as Microsoft Access), and may be invisible to the IT department or senior business data administrators. Some key data may reside in external systems maintained by service providers or business partners. To manage this you need powerful, simple, but effective models of the data structure from an enterprise viewpoint -- a set of models known as the "Enterprise Data Architecture."

This article, co-written by Richard Wiggins and originally published in the Rational Edge in February 2003 describes a new approach, based on UML, which meets the real requirements of modelling the Enterprise Data Architecture.

Architects Anonymous

Andrew Johnston — Sun, 27 Jun 2004 21:00:06 GMT

Is Architecture an Addiction?

This rather tongue in cheek piece was inspired by a "group therapy" session at the IRM/Meta group Enterprise Architecture Conference 2004.

Combining Risk Factors

Andrew Johnston — Thu, 25 Sep 2003 04:01:00 GMT

In "Waltzing with Bears" Tom DeMarco and Tim Lister introduce the very useful concept of the "Uncertainty Diagram", the probability distribution for project metrics such as delivery date, expenditure or benefit delivery. This is used, for example, to assess the likelihood of delay from a given risk.

However, they rely entirely on Monte-Carlo simulation. I believe that where the curve is defined by, or can be approximated by, a few discrete points, a relatively simple analytical solution can then be used in place of simulation.

Modelling Change in Enterprise IT

Andrew Johnston — Fri, 15 Aug 2003 01:00:00 GMT

One of the big problems in a strategic or "enterprise architecture" view of IT is how to model the change in an enterprise's IT portfolio over time. Most established modeling techniques deal with an essentially static view of the system landscape, supplemented by some modelling of the dynamics within systems. These are very poor tools if you are trying to understand how the complete set of systems, technologies and capabilities change over time. This item discusses two simple techniques which address this problem.

Characteristics of a Software Architect

Andrew Johnston — Wed, 09 Jul 2003 01:00:00 GMT

My musings on what makes a good software architect.

The Changing Role of An IT Architect

Andrew Johnston — Wed, 09 Jul 2003 03:00:00 GMT

An IT architect (for want of a generic term which isn't already terribly overloaded) takes on many different roles over time. This article discusses some of those roles, and introduces a model for the different architectural interventions in a typical project life-cycle.

The Tao of the Architect

Philippe Kruchten — Sun, 02 Mar 2003 06:49:41 GMT

Philippe Kruchten of Rational has taken the Tao of Lao-Tsu, and created a modern translation focussed on the values and attitudes which an architect should adopt. He says many things which are also said on this site, but much more beautifully.

A Square Peg, and Only Round Holes

Andrew Johnston — Wed, 09 Jul 2003 06:00:00 GMT

Musings on whether an architect can fit neatly anywhere in a traditional IT organisation hierarchy.

Technical Reference Architecture

Tim Barrett — Sat, 07 Jun 2003 06:44:36 GMT

"Technical Reference Architecture for Component Based Development and Enterprise Application Integration" by Tim Barrett of ComCor IT Solutions BV is an excellent summary of the characteristics of a strong, flexible, layered and component-based architecture, and the different classes which comprise one. (Adobe Acrobat Format)

Strategies for Flexibility

Andrew Johnston — Tue, 04 Sep 2001 08:00:00 GMT

Organisations need to protect and maximise the value of their IT assets. To protect against threats from business and technological change systems need to be flexible: able to change to support new functions, new workloads and new working environments. Flexibility does not happen by accident - it is usually the result of planning, forward thinking and adopting strategies known to enhance and encourage it.

This paper (in Adobe Acrobat Format), originally published by the CBDi Forum, presents some of those strategies.

Articles on Agile Modeling and Architecture

Andrew Johnston — Sun, 02 Mar 2003 06:43:47 GMT

Here's a list of useful articles by Scott Ambler on either the Agile Modeling or Agile Data web sites. A lot of that material is relevant, but I've picked out a few articles which are most relevant to the Agile Architect: