Richard Veryard on SOA

What's Wrong with Principles?

2010-02-02T15:07:00.000Z

Lots of people in the SOA / EA world think that principles are very important. So in this blogpost, I'm going to take a contrarian view.

(Before I start, let me cheerfully admit that I have probably done some of the things that I'm complaining about here, and if you catch me doing any of them in the future, please feel free to give me a sharp dig in the ribs.)

There are four reasons why I think principles (especially lists of principles) are overrated.

Purpose / Value - uncertainty about what the principles are supposed to achieve
Form - uncertainty about what a principle should look like
Source / Material - uncertainty about what a principle should be based upon
Process - uncertainty about how a principle should be used.

Form

Let me start with the way that principles are expressed. Lists of principles often include a mixture of assertions, commands, preferences and goals.

For example, here are a few example principles from TOGAF 9 (section 23.6).

Requirements-Based Change Only in response to business needs are changes to applications and technology made. (command)
Common Use Applications Development of applications used across the enterprise is preferred over the development of similar or duplicative applications which are only provided to a particular organization. (preference)
IT Responsibility The IT organization is responsible ... Each data element has a trustee ... (organization structure)
Data is an asset ... that has value to the enterprise ... (assertion)
Data is shared ... and accessible and defined consistently ... (goals)
Maximize Benefit to the Enterprise Information management decisions are made to provide maximum benefit to the enterprise as a whole. (wishful thinking)

And here are a few from PeaF

Strategic planning is at the heart ... Relationships are the key to understanding ... (assertion)
Adopt a service oriented approach. Adopt architecture best practice. (command)
Reusable applications will be favored. ... Reuse will be considered first. (preference)

This kind of confusion is not unique to SOA and EA, but appears in other domains as well. For example, Dave "Cynefin" Snowden's seven principles of knowledge management are generalized observations, which are true except when they aren't. He then adds four organizing principles, which are more like guidelines.

Okay, so there is some variation in the way that principles are expressed. Why should this be a problem? Because it reflects some confusion as to whether a principle is supposed to be true or useful. If we are going to accept a set of principles uncritically then maybe that doesn't matter, but if we are going to evaluate principles and apply them intelligently to a particular situation then the difference between truth and utility is rather important.

At least the TOGAF and PeaF principles have been worked on to achieve some degree of objectivity and verifiability. In other discussions on the Internet, I've found slogans like "Think Strategically, Act Tactically" being described as principles (or perhaps "guiding principles"). These might be useful reminders to the self, but not much use for governance.

Source / Material

Where do principles come from? What is their authority? If everything is supposed to be guided by a core set of principles, then we need to be confident that the principles are right.

I am particularly suspicious of principles that are supposed to be obvious or self-justifying, or are based on majority opinion. If this is something that everyone believes, it is probably either false or not worth stating at all. And anything produced by a committee of experts usually has all the interesting content leached out in order to achieve a bland generalized compromise they can all agree to.

And I am very irritated by websites such as OWASP that publish lists of "proven principles" without the slightest indication of the proof underlying any of these principles. (As far as I can see, any member of OWASP can add anything at all to the Principles page on the OWASP wiki, so the only verification consists in the fact that no other member has bothered to challenge any of them.)

TOGAF 9 (section 23.4) identifies five criteria that distinguish a good set of principles: understandable, robust, complete, consistent and stable. But in my view, the most important criterion is missing here. Principles are like policies, they should be based on robust evidence, they should be monitored for ongoing effectiveness, and subject to revision if the evidence shows they aren't working. This is one of the key differences between a science and pseudo-science - see my post Is Enterprise Architecture a science?

Process

How are principles supposed to be used? Here are some snippets.

"Guiding principles drive the IT architecture and the service model, which in turn dictate how the enterprise IT infrastructure services may be defined." (Tilak Mitra, IBM)
"The principles of service-orientation can be applied to services on an individual basis, allowing a reasonable degree of service-orientation to be achieved regardless of the approach." (soaprinciples.com)
"Each enterprise should have a set of guiding principles to inform decision-making." (Open Group SOA Sourcebook)
The purpose of these principles is not to constrain, but to provide a broad cultural framework in which work will be carried out. (PeaF)
"As projects begin to define solutions to problems they are assessed as complying with these principles or not. For a project, if all principles are complied with - no problem - no Enterprise Debt is created." (Kevin Smith, PeaF, via Linked-In).

So there is some variation here - do the principles work as hard policies, dictating and controlling a set of relevant processes and practices? Or do they merely work as soft guidelines, informing decisions and providing a basis for estimating some relevant metrics such as "enterprise debt"?

And if the principles are not so bland as to be meaningless, we should expect occasional tension between different principles. So how do we resolve conflicts between different principles?

The answers to these questions depend on our earlier discussion. If the principles can be traced to robust evidence, then they have visible authority and weight, which gives them particular force in a particular situation. But if they are only bland generalizations, then they will be ignored as soon as they become inconvenient - in which case, you might as well not bother having them at all.

Purpose / Value

So in the end, what are these kinds of principle actually worth?

Principles may be helpful in creating a common narrative about what we are trying to achieve - but if that's all they are, then we don't have to take them too seriously. Some people seem to regard principles as rules that others should follow religiously but they are free to deviate from if they feel it necessary.

To the extent that it is worth having a consistent approach to something, then a set of evidence-based policies should help achieve some degree of consistency. But most lists of principles fall way short of this idea. Instead, a so-called principle may be any of the following ...

something that is always true
something that is usually true (except when it isn't)
something that is always good (or beautiful)
something that is usually good (except when it isn't)
a random thought that someone thinks is important
something that good people believe or follow and bad people don't - a way of separating Them from Us
something that ignorant people should believe and follow, and clever people don't have to - ditto

So that's my argument against principles - they may have some limited use, but they are not strong enough to bear the weight that is commonly put on them. I know many of my friends will disagree with me, and I look forward to some robust discussion.

Towards an Enterprise Architecture of Knowledge

2010-01-20T14:02:00.001Z

A customary way of modelling an enterprise is as a collection of purposeful behaviours - functions or processes or capabilities or whatever. Each function or process or capability encapsulates some knowledge or know-how. Sometimes this knowledge will be fairly static, so the behaviour will be stable and unchanging. And sometimes the underlying knowledge will change, so the behaviour will need to be adjusted to accommodate the most up-to-date and relevant knowledge. In some areas, such as product development and marketing, there may be a strategic imperative to find and assimilate new knowledge and ideas (about market trends, technology trends, competitor trends, and so on), to build new collaborative partnerships, and to experiment with new behaviours. These areas will rely on business intelligence - not just BI tools but all the practices associated with collecting and making sense of complex information.

The overall architecture of the enterprise can then be understood in terms of how these functions or processes or capabilities interoperate to produce a viable system-of-systems. The enterprise architect must understand which of these functions are relatively stable, which of them are strategically volatile, and create appropriate mechanisms for coordinating between them. These mechanisms are essentially behavioural ones - information flows, event triggers, process flows and so on.

But if we look at the enterprise through a knowledge management lens, we will see a different kind of structure. There is a series of overlapping knowledge domains - marketing, accounting, regulatory compliance, health and safety, employment, technology - each with its own specialist terminology and ways of thinking. So there is an architectural question about how these knowledge domains (we might also call them agendas or discourses) interoperate.

In a traditional organization, these knowledge domains only come together at board level, with one or more directors speaking for each significant domain. Thus the CTO speaks for the technology domain, the HR director speaks for the employment domain, and so on. Of course all the directors will try to influence the agenda of their peers - so everyone else might be citing random snippets of evidence of technology trends, or challenging the CTO's decisions and interpretations, pushing the CTO to explain or change direction. In a well-functioning organization, this will be all done in the spirit of healthy and vigorous debate.

In a hierarchical organization, some pieces of this debate can of course be delegated - either to internal departments or to external consultants - but the directors remain responsible for putting the pieces together.

Thus the coherence and intelligence of a traditional organization depends heavily on the collective intelligence and effective functioning of the board of directors. But there are several evident problems with this approach. Firstly, we can observe many organizations that lack coherence and intelligence, so this approach is manifestly not working well enough for these organizations. Secondly, we may assume that the board of directors has a finite reasoning capacity - however clever they are as individuals, and however well-bonded they are as a team - and that they will not always be able to keep up with the increasing complexity and volatility of the business environment. And thirdly, we can see a lack of joined-up intelligence at the lower levels of the organization.

Some people think there is a different way of integrating an enterprise. If we can identify a single universal knowledge domain, then we can use this knowledge domain to join up everything else. For example, in a financially driven organization, management accounting acts as a unifying language across all functions. Conversely, many enterprise architects regard their favourite EA framework as providing such a unifying language. Although there have certainly been very large organizations that have been managed according to a single unifying principle (such as GEC under Lord Weinstock), the fact that there are several specialisms each wanting to be top dog helps to explain why this doesn't work very often.

A third idea would be a bottom-up Enterprise 2.0 swarm intelligence, in which a satisfactory resolution to all difficult issues emerges from uncontrolled debate and "sharing" across the organization. To read some of the material on the internet, it's tempting to believe that this is how some of the really cool organizations in Silicon Valley are organized. But even if this were true, there is still a need for some kind of architecture and governance, for example, providing suitable platforms and pathways for constructive and reasonably efficient problem-solving. Kevin Kelly, whose book Out of Control introduced me and many other people to the huge potential of bottom-up intelligence, is careful to warn that the bottom is not enough.

Which brings me to the fourth idea - regarding the coherence and intelligence of the knowledge-bearing organization as a real architectural issue. Not the kind of structure that enterprise architects are accustomed to modelling or managing perhaps, but this structure may have a significant effect on the business outcomes of the organization. Isn't this something enterprise architects should be interested in?

Intelligent Knowledge Management

2010-01-15T15:41:00.004Z

@snowded has started a series of blogposts about knowledge sharing across silos (part one, part two). He starts by identifying a number of different types of knowledge that (some people think) it might be valuable to share.

Data - avoiding duplication between data stores - for example consolidating all health records in a single database

Information - ensuing all relevant information is available to support critical decisions and actions - see for example my post on Joined-up Healthcare

Transactions - a sense that the citizen or customer has to navigate tortuous pathways through the bureaucracy

Practice - good practice developed in one area is not appreciated or understood elsewhere

Ideas - innovative combination and variation of issues, ideas, solutions and problems from within different silos

Obviously a mixed bunch of stuff here, and it seems very confusing to lump all of this under the heading of "knowledge management". As Dave points out, some of these look more like information flow. However, suppliers of knowledge management products and services (consultants and tool vendors) typically want to find as many different applications for their stuff as they can. Knowledge is therefore presented as a way of achieving a bunch of different things.

more efficient information systems and working procedures
improved decisions
more effective and convenient customer service
faster, more efficient and more consistent innovation
greater cooperation and collaboration

Underlying the knowledge management agenda are a number of (usually) unexamined assumptions

most of the knowledge we need already exists in people's heads - we just have to get it out
knowledge is additive - the more knowledge we can "capture" or "extract" the better
explicit knowledge is better than implicit or tacit knowledge - codification is a "good thing"

Thus knowledge management becomes a combination of documentation/codification and librarianship, plus whatever persuasion or coercion may be necessary to encourage people to participate in this game. Added to this is a kind of open-ended information retrieval, as exemplified by IBM's Lotus Knows campaign. (See my comments here.)

Undoubtedly there are some situations where this kind of thing can deliver direct value to the organization - for example in pharmaceuticals, where any delays in the assembly of regulatory information for a new drug can cost millions of dollars in lost revenue. But is this really knowledge management, or just a sophisticated form of information management?

An alternative approach to knowledge management is to leave it in people's heads, but then to provide knowledge maps that tell everyone whom to ask. The trouble with this approach is that the genuine experts often keep their heads down, for fear of being swamped with enquiries from around the globe, while attention-seekers use this as an opportunity to promote themselves. Thus questions of motivation and excellence must be addressed, and knowledge management becomes a branch of Human Resources.

Meanwhile, I've been reading a couple of older critiques of the 'knowledge management' industry.

Erik Bergman, When bad things happen to good ideas (CIO, May 2001) via @rotkapchen
T.D. Wilson, The nonsense of 'knowledge management' (Information Research, October 2002) via @oscarberg

Although these papers predate the recent explosion of web-based knowledge management tools (including Enterprise 2.0, however that is defined), the issues raised in these two papers largely remain unaddressed. Bergman points out that knowledge management is often regarded as a technological initiative. He mentions a number of very expensive white elephants, which failed largely because they failed to address the real business objectives or the real cultural and working practices of the organization. Wilson raises some more fundamental issues, suggesting that the knowledge management agenda is muddled and self-serving, and that Polyani's original concept of tacit knowledge has been grossly misinterpreted in order to promote an utopian fantasy of the manageability of the human mind.

So what (if anything) can we rescue from this mess?

Well we might start by shifting from "knowledge as a resource" to "knowledge as a process", as Gil Ariely argues in his paper Knowledge Management as a Methodology towards Intellectual Capital (Sept 2003, pdf). See also Interview with George Siemens (Dec 2009). Unfortunately, this phrase was long ago coopted by solution providers - for example in their paper Knowledge Asset Management, (June 2001), Ron Young and Gregoris N. Mentzas describe what they claim to be "one of the world’s first truly holistic and integrated knowledge management solutions". Holistic, huh?

Perhaps knowledge management shouldn't be about grabbing and hoarding stuff but about deploying it intelligently. This points us towards concepts like Evidence-Based Management, where knowledge is collected, analysed and deployed within a management loop. Consultancies promoting this and similar concepts include Pfeffer's and Sutton's Evidence-Based Management, as well as Management Epistemology. and the Knowledge Management Consortium - see for example the KMCI paper Corporate Epistemology (November 2003, pdf).

This kind of approach shifts the emphasis from knowledge sharing to knowledge embedding - grounding the work in the best available and critically evaluated knowledge, as well as actively seeking well-grounded knowledge to support organizational learning. Obviously collaboration is important here, but there is a distinction between collaborating-in-the-work (for example shared responsibility for decisions) and collaborating-in-the-knowledge (for example, shared responsibility for collecting and interpreting intelligence, connecting the dots). This distinction builds upon the RAEW technique we have long used for analysing organizations. A key question here is the relationship between decision and intelligence - how closely the expertise and authority should be coupled/aligned to the work itself.

What characterizes the intelligent organization is not just having more knowledge than its competitors, or even doing better at sharing the knowledge it has, but how effectively it invests its entire knowledge capital to increase its own viability and survival. This is so not a technology question, or even a best-practice question, but a strategic question. But you already knew that, didn't you?

Business Design Choices

2010-01-11T11:41:00.002Z

The SOA Consortium (part of the OMG) has published a Business Architecture Paper, edited by Brenda Michelson. The paper presents the following case for formalized Business Architecture under IT leadership.

1. Business architecture is a critical input to IT planning, technology architecture and business solution delivery.

2. Technology trends and IT capabilities influence business design choices in the realms of capabilities, value chains, processes, and channels.

3. Business architecture provides the mechanism to clearly illuminate how strategy, processes, business structure and staff can best be utilized to deliver reliable and cost effective operations. With this clarity business can enable new functions and services, with the right resources and technology, effectively and efficiently.

4. Business architecture can help organizations analyze key value chains. Organizations can eliminate duplicate operations, processes and technologies across business units and functions. Business architecture provides the methods to analyze and plan how the organization should morph its structure, processes, technology and staff.

5. The most likely origin of a business architecture practice is within IT.

It is difficult to see anything here that wasn't included in the Business Systems Planning and Business Engineering methodologies I first saw in the 1980s - for example, as a front-end to such methodologies as Information Engineering. Nowadays, Michael Porter's value chain concept is taught in high school business studies; so unless business managers are really thick, they are not going to need help from IT folk to "analyse key value chains".

Actually, there are some more advanced business design choices where business architecture has a useful role to play, and where some IT people might have some relevant aptitude, but the SOA Consortium doesn't mention any of these. I see the real challenge for business architecture being to appreciate the economic and social impact of structure. And by structure, I don't just mean the kind of line-and-box diagram or simplistic block diagram that most so-called architects produce, whether in Powerpoint or some cheap modelling tool, but something that reflects the real complexity of the business.

So what are these "business design choices" then? Well, this is a topic I've been talking about for many years, going back to my 2001 book on the Component-Based Business. Here are some critical ones, all of which I've discussed on this blog before, as well as in articles for the CBDI Journal.

1. Coupling. Which "components" of the business need to be closely aligned (tightly coupled) and which "components" should be autonomous (loosely coupled)? How is an enterprise governed as a system-of-systems? How do we reason about the properties of the whole enterprise?

2. Differentiation and Integration. Where does standardization make sense, and where should requisite variety be deployed? This question goes back to the work of Lawrence and Lorsch (see summary here), and has recently been rediscovered (in slightly different terms) in the book Enterprise Architecture and Strategy.

Operating Model Quadrants (Adapted by Clive Finkelstein from Figure 2.3 of “Enterprise Architecture as Strategy”) - Source The Enterprise Newsletter #38.

3. Business as a Platform. A business provides a set of services to its customers. There is a strategic choice between a high-volume generalized platform, which takes a small slice of a large amount of activity, and a lower-volume specialized platform, which takes a much larger slice of a relatively smaller amount of activity. So for example, how does a telecoms company create value-adding services on top of the basic calling services (which are getting continually cheaper, thus eroding core revenue) without losing the economics of scale.

4. Edge Organization. What are the structural implications of turning your business upside-down and inside-out to address the dynamics of the demand environment?

5. Viable Business. What structures are required to support the intelligent organization?

So there is a set of really interesting and important issues here, which some of the smarter business people in the more complex industry sectors (aerospace, telecoms) are already trying to tackle. There are some new and emerging techniques for tackling these complex problems, including Asymmetric Design, as well as some older but underused techniques. However, I don't yet see much evidence of enterprise architects stepping up to the real challenges of business architecture.

Alignment as a Percentage

2009-12-31T15:16:00.000Z

Many people perceive value in something called Business-IT alignment, and argue that we should invest in tools and methods that increase this alignment.

Does the word "alignment" actually mean anything concrete, or is it just a vague metaphor, as I suggested in my previous post Alignment - Science or Pseudoscience? Here's my challenge to the alignment brigade: I'm not interested in "business-IT alignment" unless it can be expressed as a percentage.

Let's imagine we can define a scale from 0% (no alignment whatsoever) to 100% (total alignment). I guess the two extremes would be practically impossible, but 80% would be significantly better than 40%. So we could suppose that any investment that increased alignment from 40% to 80% would be worth considering, if the costs and risks were acceptable.

But at present, we don't have an agreed scale. People sometimes talk as if alignment was some metaphysical goal (like love), rather than a practical manageable outcome. But while I accept the importance of relationships and trust within organizations, I don't see this as part of the concept of alignment.

And I'm not convinced that Business-IT alignment is necessarily a good thing anyway. As I said at a CBDI Forum meeting back in April 2000,

"There are some organizations where business and IT are perfectly aligned: they are standing side by side, with their heads in the same sand, aligned in a shared sense of complacency. In other organizations, the opposite is true. It is often in the dynamic, progressive organizations where business and IT are furthest apart." [CBDI Journal "Interact", May 2000]

Furthermore, why is "alignment" only a question for IT. @malcolmlowe comments "interesting how no-one measures business-finance alignment as %. It's just part of bus". And what about HR or other support functions?

However if we had a scale I believe it could be a useful management tool, provided that the results were properly interpreted. Given how many CIOs worry about the "alignment" problem, some of them might want to assess and benchmark the alignment percentage from time to time, to see if it is going up or down. As long as it is clearly understood that there are many things that IT needs to be "aligned" with, not just a simplistic notion of "the business".

Alignment - Science or Pseudoscience?

2009-12-30T16:03:00.001Z

@RSessions claims that "Simplification occurs when the IT partitions align with the business partitions". But what does alignment mean?

Roger defines alignment as "a measure of how well two patterns overlay on top of each other". But this definition only works between two patterns that already occupy the same space.

For example, I think I know what it means to say that the furniture is aligned with the walls of the room, because I can measure this fact with geometrical instruments (ruler, set square), but if someone says that the colour of the furniture is aligned with the sensibilities of the owner, I can only make sense of this as a vague metaphor rather than a precise measurable fact.

The fallacy of astrology does not lie in the detailed analysis of alignments between the planets and stars - which after all occupy the same astronomical space - but in the notion that the movements of the planets against the stars are somehow aligned with the patterns of human affairs on earth. (And the reason this counts as pseudo-science is that the detailed correlation between sky and earth relies on an interpretation by an astrologer.)

When people talk about business-IT alignment, I can only make sense of this as a metaphor. I am not aware of a robust and meaningful formalization that would permit both business and IT to occupy the same geometrical space, and I can't see the point of inventing one.

All we need is a mapping between two patterns occupying different spaces. An alignment is a special kind of mapping within a fixed geometrical space. If we can't define a fixed geometrical space, then I regard the concept of "alignment" is a misleading metaphor, introducing unnecessary complication. I prefer the general concept of mapping to the false metaphor of alignment.

@aleksb6 objects that my position is "true iff two patterns to map. reality is a M2M relationship, so 'alignment' is not a complication, it's a necessity!" He continues "btw, isn't mapping two patterns just another instance of point-to-point thinking? I thought we wanted to discourage that!"

If alignment doesn't make sense between two things, I can't see that it makes any more sense between three or more things. The desired outcome is a set of structure-preserving mappings between as many things as we need to coordinate. That doesn't mean we can or should design each mapping separately. In all but the most trivial situations, however skilfully we decompose a large problem into subproblems, there is always some coordination (juggling) left to do.

Here's the bottom line. If assertions about business-IT alignment are to mean anything at all, then you have to have a way of looking at a lump of business and a lump of IT and say whether they are aligned or not, and if so how much.

Of course, if you believe you have a modelling language that can express both business and IT, then you might think all you have to do to find out if business and IT are aligned is to compare two models. But this turns out to a circular procedure, because the modelling languages are themselves justified by the claim that they promote business-IT alignment, so we cannot use the modelling languages themselves to prove that business-IT alignment has been achieved. There has to be some point of reference outside the modelling languages.

People keep telling me "alignment" is important, but they can only define it as a woolly and subjective metaphor. So if we stop worrying about "alignment", and talk instead about the various multi-dimensional mappings between a complex system of systems and a complex set of business requirements, we can concentrate on what is objectively important.

And leave the concept of "alignment" for the astrologers. All together now ...

When the Moon is in the 7th house and Jupiter aligns with Mars
Then peace will guide the planets and love will steer the stars.

Beyond the Predictive Enterprise

2009-12-21T01:39:00.001Z

@ebizQ James Taylor says you (or more likely your organization) should be striving to become a predictive enterprise. His definition comes from an old SPSS presentation.

Derives maximum value from its data assets
Understands its business by gaining deep insight
Leverages advanced analytics to predict outcomes
Turns this knowledge into action to optimize decision making across all areas of its operations

But prediction forms part of an important learning loop. We make predictions in order to take better actions. There is no point making predictions about customer behaviour unless this prediction allows you to create some value for the customer.

Secondly, we take actions with uncertain outcomes, in order to improve our future ability to predict. Received wisdom may tell us that the “best” colour packaging for a given product is blue, but an inquisitive organization will continually experiment with different packaging rather than settle for a “best practice” solution.

Thirdly, intelligent behaviour allows for its own likely effects. For example, if an intelligent organization drops its prices, it has already considered how (and how quickly) competitors might respond. (In contrast, a stupid organization predicts the outcome of a marketing offensive as if they don't expect other players to retaliate.)

So instead of predictive, I prefer the term anticipative or anticipatory. Anticipation means not passively predicting the future, but taking action in advance of the future. Robert Rosen defined an anticipatory system as "containing a predictive model of itself and/or its environment, which allows it to change state at an instant in accord with the model's predictions pertaining to a latter instant". Thus anticipatory contains a degree of self-awareness and self-embodiment, which allows proactive and agile action, not merely disinterested forecasting. This is an essential aspect of what I call Organizational Intelligence.

EA maturity models

2009-12-15T16:15:00.000Z

Following the popularity of SOA maturity models, a number of EA maturity models have started to appear.

However, if the purpose of this kind of exercise is to assess and improve the effectiveness of an Enterprise Architecture program, then I think it's more useful to think of these models as Excellence models rather than Readiness or Maturity models, and to take inspiration not just from the SEI CMMI but also from excellence models such as Baldrige and EFQM.

There are two important points about these excellence models. Firstly, they don't only look at the capabilities but also at the outcomes produced by these capabilities. And secondly, they provide a systematic framework for developing a customized capability model for a particular enterprise. We shouldn't expect people to invest in specific EA capabilities just because some EA guru thinks it's a good idea, or just because lots of other organizations have adopted this as a "best practice", but ultimately because this capability can be demonstrated to produce the desired effects in this particular enterprise. Whereas an immature organization may have to take some of this kind of thing on trust, a mature organization should be striving for EA excellence, which means that every EA activity is tied to results.

Incidentally, when I have talked to people about Adoption and Excellence models in the context of SOA (in my work with the CBDI Forum), I have perceived a little resistance to the word "excellence". Some people seem to interpret it as meaning quality for its own sake. But in Baldrige and EFQM, excellence means value-for-money, doing exactly those things that contribute to the short-term and longer-term goals of the enterprise and its customers. So perhaps we could call it a Cost-Effectiveness model instead.

So I think any EA assessment should include three vital questions. What does EA cost - not just the architects' own time but also the time of developers, users and other stakeholders in participating in EA activities and complying with EA edicts? Secondly, what added-value is EA producing for the enterprise? And thirdly, what is EA doing to monitor and control the answers to these questions?

To get good answers to these questions, we clearly shouldn't just ask the architects; interviews should involve developers, users and other stakeholders: thus we end up with a 360-degree assessment.

Business Rule Concepts

2009-12-03T00:02:00.000Z

by Ronald G Ross. Third Edition: Getting to the Point of Knowledge. Business Rule Solutions 2009.
ISBN 0-941049-07-8

A copy of this book arrived in the post this week. I don't know who sent it but I assume it's a review copy from the publishers. (In which case, the publishers have chosen not to implement the standard Cover-Note role. Perhaps they have implemented the If-Anyone-Reviews-It-We'll-Probably-See-Something-On-Twitter rule instead.)

As the title indicates, the book provides a detailed account of the concept of Business Rule, and appears to offer some elements of a methodology for doing something or other. My normal procedure for reviewing such books is to apply the Four Causes.

Final Cause - what's the purpose, for whom
Material Cause - what is the stuff that is being worked on
Efficient Cause - who does it, and how
Formal Cause - what's the conceptual framework

Most of the book is taken up with a conceptual framework for business rules. What does a business rule look like, and how can different kinds of business rule be classified. So that's the Formal Cause.

Ross argues that just as the human body is composed of bones (structure), muscles (power) and nervous system (control), so business is composed of factual knowledge (structure), processes (power) and business rules (control), expressed as nouns and verbs.

To me, this looks like a huge simplification. For a start, in order for the human body to do anything useful, the muscles need energy. The biological control system doesn't only control the muscles, it also controls the respiration system - delivering stored energy to the muscles. And for a business to be viable, we generally need to pay attention to the adverbs as well as the nouns and verbs. But in order to consider whether Ross's simplification is useful for some purpose, we need to look at the other three causes.

A business rule is supposed to describe the business. This suggests that there is some stuff (let's call it the implicit logic of the business) from which properly constituted rules can be elicited - the Material Cause. Rules are correct and complete if they accurately and fully capture the AS-IS or TO-BE logic of the business. (Ross tells us little about verification and validation, except to say that it's easier if the rules are held in some kind of repository.)

But which logic? Ross doesn't want to get sucked into expert systems or artificial intelligence (p 23). He argues that the point of establishing business rules is to give you more time for strategic and tactical thinking, problem solving and other stuff (pp 4-5). I infer from this that he is not trying to elicit the logic of these higher-level activities: his methodology concentrates on what he calls Operational Decisions.

The book also contains a number of rules about rules. For example

A rule saying there shall be no rule restricting the age at which a customer can open an account (p 92). This kind of rule is important in any organization where different units can define some of their own rules.
A rule saying that the cheapest rule should be executed first (p 128). This kind of rule is important for balancing the internal efficiency of a system with the external customer experience, and assumes we have some way of associating different kinds of cost with a rule. (The cheapest rule from the programmer's point of view may be very expensive in terms of customer satisfaction and lost business.)

These rules-about-rules are not formalized, and so I'm presuming they are not within the scope of the business rules methodology. However, some important rules-about-rules are stated in the Business Rules Manifesto (p 143-144). For example

6.3 - A business rule system must always be able to explain the reasoning by which it arrives at conclusions or takes action.

6.4 - Rules are based on truth values. How a rule's truth value is determined or maintained is hidden from users.I suspect that these two rules conflict with each another. Unfortunately, I can't verify this suspicion, because the vocabulary used to express these business rules is not coordinated (see p 108). Perhaps Mr Ross should eat his own fish-fingers.

Now let's look at the Efficient Cause. The book seems to be aimed at a business analyst trying to specify the requirements for a system - either a completely automated system or what Ross calls a Really Smart System, which may include people making smart operational decisions with the support of rules that are rendered as guidance messages. However, I didn't get a very clear picture of how Business Rules might fit into a normal systems development methodology. Indeed, the back cover blurb advertises "a common-sense approach to solving today's operational business problems", but I could find nothing in the book that looked like a real business problem (as opposed to fairly routine business requirements.)

(To be fair, I should say that Ross's separation between facts, processes and rules seems consistent with some of the business modelling ideas I have published through the CBDI Forum, so I'm not saying it's wrong, merely that it may not be the whole story.)

Last but not least, the Final Cause. What is the value of this methodology, to whom? Ross asserts that the explicit separation of factual knowledge, processes and business rules produces more effective solutions, and therefore more successful (effective, adaptive) business behaviour (p 5). His approach has the "potential for closing the requirements gap between business people and IT" (p 28). And he asserts that for some class of crucial everyday decisions "you can capture all the business rules and make precise or optimal decisions" (p 31).

Ross doesn't provide any evidence for these assertions, but he doesn't have to, because they are "proven". Humph.

So where does that four-cause analysis leave us? What Ross gives us is a lot of detailed discussion about business rules and how they can be analysed. If you are already sold on the concept of business rules, and possibly interested in using business rules to design a layer of capability services that are decoupled from the entity layer and the process layer, then you may find the book very helpful.

Why have I devoted a long blogpost to discussing this book? Because I think it's important for design methodologies to be presented in a balanced way - covering all four causes.. Ross's book is typical of many such books - fairly sound on the Formal Cause, not so strong on the other three causes - and I think this imbalance helps to explain the patchy adoption of such methodologies.

Complexity and Power

2009-11-28T13:14:00.001Z

@RSessions kindly gave me a quick overview of his SIP methodology, and how he calculates the complexity of a system of systems, based on the number of elements in each system and the number of connections between systems. The internal complexity of each system increases in a non-linear manner with the number of elements, and the external complexity increases with the number of connections between the systems, so the trick is to find a structure that optimizes the overall complexity.

Obviously we have to be clear as to what counts as an element (for example functions), and what counts as a connection. Using the SIP lens, it is possible to see how certain architectural styles (including those popular in the SOA world, such as hub-and-spoke or layered) only deliver simplicity (and the benefits of simplicity) if we can assume that only certain kinds of connection are significant. Roger's view is that this assumption is unwarranted and invalid.

In general, the so-called functional requirements are associated with the elements and the logical connections between them. In my view, architects also need to pay attention to the nature of the connections (coupling) because these will have important consequences on the structure and behaviour of the system as a whole. For example, synchronous versus asynchronous. At present, Roger's complexity calculations don't differentiate between different kinds of connection, so it would be interesting to investigate the costs and risks associated with different kinds of connections, to see how much difference it could make.

Roger's primary interest is in IT systems, but the same principles would appear to apply to processes and organizations. If you are running a factory, you have an architectural choice about the connection between say the moulding shop and the paint shop. With an asynchronous flow you have two loosely coupled operations separated by a buffer of work-in-progress; with a synchronous flow you have two tightly-coupled operations connected on a just-in-time basis. The former is a lot easier to manage, but it has an overhead in terms of inventory cost, storage cost, increased elapsed time, slower response to changes in demand, and so on. The latter may be more efficient under certain conditions, but it can be more volatile and the impact is much greater when something goes wrong anywhere in the process.

Intuitively, there seems to be a difference in complexity between these two solutions. The first is simpler, because the connection between the two systems is weaker; the second is more complex. With greater complexity comes greater power but also greater risk. Surely this is exactly the kind of architectural trade-off that enterprise architects should be qualified to consider. Roger's SIP methodology does give the architect a very simple lens to try and understand system-of-system complexity. Not everyone agrees with Roger's definition of complexity, and we can find some radically different notions of complexity for example in the Cynefin world, but at least Roger is raising some important issues. The EA world certainly needs to pay a lot more attention to questions like these.

Is Enterprise Architecture a Science? Part 2

2009-11-28T08:55:00.096Z

@RSessions was in London this week, so I sat down with him to continue our previous discussion Is Enterprise Architecture a Science?

The first question to address is - which enterprise architecture are we talking about? I think we both agree that there are some activities within the EA world that look more like religion or mediaeval scholastic philosophy than empirically verifiable science.

For example, in his post What's Right with the Zachman Framework, Grant Czerepak states that "the architectural metaphor conceals what the six perspectives are actually about: Entities, Relationships, Attributes, Constraints, Definitions and Manipulations". And referring to the Kipling-Zachman lens, Grant claims that "the interrogatives have a foundation that goes back over three thousand years across every human culture". (In a separate post, he says It's not Aristotle's fault, it's your fault.)

Lots of EA frameworks are essentially abstract classification schemes that start from an abstract ontological argument ("obviously all businesses are made of objects" or "obviously all processes are made up of nouns and verbs") and make assertions that are not amenable to empirical verification.

Roger's SIP methodology is at least based on an empirically testable (and quantified) hypothesis. That a system of systems with such-and-such measurable structural qualities (in terms of Roger's definition of complexity) will have such-and-such predictable costs. So this provides the basis for a scientifically-grounded engineering practice. I think SIP methodology has a reasonable claim to be scientifically grounded: it can be evaluated not just on whether its prescriptions are practical, cost-effective and useful, but also whether its predictions are true. (Incidentally, I think it would be interesting to compare SIP with Christopher Alexander's early book Notes on the Synthesis of Form. This contains detailed and mathematically grounded work on architectural complexity: although the software gurus who developed structured methods in the 1970s were aware of Alexander's book, they left out most of the difficult detail.)

I think there is a further step before EA could ever dream of becoming a fully empirical science, and this would involve large-scale collection and analysis of empirical data, so that there would be a closed loop between theory and practice, connecting structure and value. In order to achieve this, we should need the active participation of some of the more powerful players in the EA game - the large consultancies and above all the key government agencies that govern IT expenditure. (You know who you are.) At the moment, there is little sign that these organizations are seriously interested in any game-changing innovation. (Roger and I should be delighted to talk to representatives of these organizations, please contact us.)

SOA Planning and Subsidiarity

2009-11-19T11:55:00.001Z

The principle of subsidiarity states that central planning only applies to those things that require central planning.

Applying this principle during the planning process means finding an appropriate level of consistency and sharing of policy and services and infrastructure. The concept of subsidiarity refers to the scoping level at which a given set of actions and outcomes are coordinated. Which aspects can be (or must be) determined locally, and which aspects can be determined centrally? Which aspects are managed at department level, or at sector level, or at national level? This calls for architectural thinking about management.

In the early phases of SOA adoption, the primary challenge may be to constrain departmental autonomy over some key issues. However, in the later phases of SOA adoption, SOA-based thinking can be used progressively to decouple enterprise activity. Business transformation may sometimes reduce the need for tight synchronization between different processes, while technology and common standards help reduce the interoperability risks. Such changes may have the effect of empowering a degree of greater local autonomy and differentiation (diversity) against a common platform of shared services. This outcome becomes progressively available as the SOA adoption becomes more mature.

Knowledge and Architecture

2009-11-17T19:22:00.004Z

@EnterprisingA (Jon Ayre) suggests an #EAMantra "If you know it, add it to your architecture. If you think you know it, add that too."

My immediate objection to this rule was that it seems to turn the architecture into a kind of brain dump. A good architect knows many things, and if all these items of knowledge go indiscriminately into the architecture, the architecture gets rather cluttered.

@EnterprisingA 's first defence was to say that "the quality of a brain dump depends on the quality of the brain from which the dump comes". And of course only the architect gets to deposit knowledge into the architecture (otherwise there would be a free-for-all).

Unfortunately, the brilliance of the architect is no guarantee that the architect's thoughts all satisfy some meaningful criterion of quality. Intelligence doesn't necessarily mean always having better-quality knowledge, but it should mean having a better capability for processing and developing and filtering and revising knowledge. And as Dumbledore says, "Being - forgive me - rather cleverer than most men, my mistakes tend to be correspondingly huger".

So the architect's knowledge may well get into the architecture eventually, but it doesn't need to go straight in. But @EnterprisingA is (quite reasonably) worried about not writing stuff down. "Keeping it in your head until you are absolutely certain is a good way of never being certain (or being challenged)." "I'd like you to review the arch but it's not certain til it's reviewed so I haven't drawn it so you can't review it..." Of course that's true, but I don't agree that there are only two places to keep knowledge - either in your head or in the architecture. @rgechristie suggests a third possibility - an interim area for in progress work before it hits "the architecture" (that still needs to be accessible by all developers and consumers of the architecture).

***

My second objection was that the architect has a lot of knowledge that doesn't really belong in the architecture at all. There is a strong temptation to put too much into an architecture, and the architectural documents can easily get bloated with miscellaneous material that the architects imagine might be useful to developers and others. (Methodology documents are subject to the same tendency.) For example, the architect may review a design document, may spot a particularly egregious design flaw, and decide to add a rule into the architecture, so that the architecture evolves into a general-purpose design handbook. And if every pearl of wisdom spoken by the architect has to be preserved in "The Architecture", you end up with "The Baroque" - a highly decorated and convoluted style.

It is of course possible to slim down an architecture with too much detail, but in practice this doesn't happen as often as it should. It is much easier to add material to a document than to subtract it, so over time the documents get longer and harder to read.

***

Following our debate on Twitter, @EnterprisingA reformulates his mantra to specify architectural decisions only. "If you know it is the right architectural choice, add it to your architecture. If you think you know it is, add that too." Surely I can't disagree with the proposition that "architectural decisions should be in the architecture", can I?

Actually I do. Sometimes the best architectural decision is to leave something out of the architecture, to leave something deliberately open and unspecified, underdetermined rather than overdetermined. This is a principle of just-enough architecture, or Zen architecture. Sometimes less is more.

How Dashboards Work

2009-11-10T09:34:00.000Z

There are three ways of understanding a dashboard.

Symbolic. A dashboard simplifies and codifies What-Is-Going-On.

Imaginary. A dashboard creates an illusion of an accurate and comprehensive picture of What-Is-Going-On.

Real. The dashboard always falls short of representing What-Is-Going-On. There is always a shadow - something left over that eludes simple capture and representation, often remaining invisible or inaccessible.

Dashboard design and development often focuses on the symbolic - defining the contents of the dashboard as a aggregation of services and data feeds, defining the events that are to be displayed (for example as warning lights), and setting simple policies that set thresholds of attention for predictable events.

Some people seem to view this as a key element of systems thinking. For example Bas de Baar, who blogs under the name "Project Shrink", identifies Systems Thinking As A Technique To Find Project Problems and claims "If I have the right metrics I can ignore everything around me and focus just on the dashboard." He appears to justify this claim by comparing project managers with fish (Swimming Upstream The Information Flow). Fish may be reasonably well adapted to many environments, but they cannot deal with the complex threats posed by the much more intelligent dolphin, as I pointed out in my blog on Lean versus Complex.
However, an emphasis on the dashboard as a simple collection of metrics overlooks the way the dashboard is used within a socio-technical system. Isaac Asimov wrote a story called Reason in which a robot controlled a dashboard perfectly, while refusing to believe in any system beyond the dashboard, but of course that's science fiction.

If we imagine that the purpose of a dashboard is to support prompt and appropriate action in a wide range of normal and abnormal operating conditions, then it should support as much (human, organizational) intelligence as is required to maintain the viability and safety of the system in a given complex environment.

One of the lessons from the Three Mile Island disaster was that when something goes seriously wrong, all the red lights on the dashboard start flashing at the same time, and unless the people in charge of the system have some way of making sense of the emergency (emerging situation), they may make things worse rather than better. (Deming calls this tampering or meddling.)

The safe operation of the nuclear power plant is not just about the design of the dashboard, or about the training of the operators, but about the whole system producing good outcomes in all circumstances. In the Springfield Nuclear Plant, the risk comes not just from idiot operators (Homer Simpson) but from corrupt managers (Montgomery Burns).

Many dashboards are designed merely to aggregate and push information into a social system that the dashboard designer doesn't bother to understand. Prior to Three Mile Island, this was often true even in safety-critical situations. I trust that the safety-critical world has now learned this lesson, but dashboards in other contexts may not be so conscientiously designed and tested.

In management information systems, a dashboard focuses executive attention onto specific aspects of the business. But there seems to be an important difference between a random collection of KPIs or guiding ratios, and a joined-up view that helps the executive reason about the business as a whole system. The standard dashboard is lacking at least two elements of organizational intelligence: Sense-Making (helping the executive see how the different items are interrelated) and Double-Loop Learning (not just getting better at meeting fixed targets, but setting more appropriate targets).

So a dashboard needs to be designed to perform a clear function within an action-based command and control system, rather than merely a simplistic reporting function.

However, there are two traps here. Firstly the hubris of the designer, imagining that a complete understanding of a complex system is possible, or expecting to produce a perfect shadow-free dashboard. Secondly, the blinkered vision of the operator, staring at the dashboard and failing to look out of the window.

In any case, management-by-dashboard seems a pretty unauthentic and disengaged way of running a company. Perhaps it is ironic that HP, one of the leading technology companies of our time, boasts its co-founder Dave Packard as one of the earliest modern practitioners of the opposite technique - "management by walking around". (HP Timeline 1940s)

This blog is an edited version of my contributions to a discussion on the Lenscraft Linked-In group. Thanks to Aidan Ward, Geoff Elliott and Hans Lodder for their stimulating comments.

Enterprise Architecture and Economics

2009-11-02T16:11:00.000Z

@jpmorgenthal lays out the reason why enterprise architects should study economics.

But he is not talking about your grandmother's economics. He is particularly impressed by the precise evidence-based insights of the Freakonomics project, and points out the failures of established "best practice" in the economics field.

"Stephen Levitt undermines what many other Economists, experts and pundits before him rolled out as proven facts and only due to his keen mind and his approach to formulating the problem domain was he able to uncover that which his peers could not."

JP also points out the need for evidence-based decisions to be grounded in the specifics of the problem domain.

"It is the Enterprise Architect's role to ensure that the selection and approaches toward development of systems are sound relative to their business, not just other businesses. Moreover, where decisions are based on the work of other businesses' success, those successes need to be properly vetted to ensure that there is enough commonality between efforts, such that you could make the leap that your business will see similar results. Finally, assumptions and theories need to be tested by properly identifying the variables that need to remain static and then comparing; in essence normalizing the question to be homogenous in all situations."

"You cannot demonstrate the value of Enterprise Architecture if you cannot monetize or enumerate the value of all possible choices relative to the choices that are being recommended or those that have been made. Moreover, it's critical that these analyses are carried out over enough time that short-term wins don't supersede long-term potential gains."

I completely agree with JP's demands, but I wonder how many economics courses you could learn these skills from.

JP concludes

"It is here that Enterprise Architects, especially those we call Chief Architects, truly show their mettle. It is their experience, coupled with the ability to focus on the right set of variables, understanding the impact of change of those variables and being able to communicate that in a way that allows the business to make effective business decisions, which sets top notch practioners apart from Sr. Software Engineers that the organization placated with a title to keep them happy so they wouldn't leave."

So what is the combined market share of EAs with these skills? A tiny fraction of a percent? And what would be the implications of this percentage on IT decisions?

Ecosystem SOA

2009-10-29T13:47:00.000Z

The SOA world is finally catching up with some of the ecosystem ideas that I published in my 2001 book on the Component-Based Business, and developed further in several articles and presentations for the CBDI Forum over a number of years.

Supply and Fit (September 2000)
Identifying Components 2 (October 2001)
Opportunities in the Insurance Ecosystem (December 2004)
Business Strategy Planning for the Service Economy (May 2006)

The biological approach to creating business and software services is radically different to the solution-driven approach, and is based on biological and ecological metaphors.

First we identify an ecosystem, which may contain both human users and existing artefacts.
Then we identify services that would be meaningful and viable in this ecosystem.
Then we procure devices that enable the release and delivery of these services into the ecosystem.

I previously defined Three Types of Requirements Engineering, and we can map these onto different styles of SOA.

Solution-Driven (Specific)	Solution-Driven (General)	Evolution-Driven
Identify Business Problem Identify "Users" Negotiate Requirements Define Solution	Identify Domain Identify Domain Experts Define Requirements Design Solution Kit	Identify Ecosystem Identify Services Procure and Release Devices
Experimental SOA	Enterprise SOA	Ecosystem SOA

(Some people use the term Web Oriented Architecture (WOA) for what I'm calling Ecosystem SOA.)

If we regard these as phases of maturity, then we can have a straightforward roadmap from left to right, as in for example the CBDI SOA Roadmap. However, some organizations may need to tackle these styles of SOA in parallel rather than in sequence.

A service portfolio plan for Enterprise SOA can be based on an enterprise model that identifies the capabilities of the enterprise and clusters these into domains. In the CBDI Forum's SAE methodology, the domains are classified as Core and Contextual according to a matrix derived from Geoffrey Moore. (Note how the domains migrate around the matrix over time.) See for example my blogpost Tesco outsources core eCommerce.

A similar model, derived from Amin and Cohendet's book Architecture of Knowledge, explicitly describes Core and Periphery in terms of knowledge intensity. In other words, the reason something is classified as Core is because it encapsulates some important (strategic) knowledge.

For example, an insurance company knows more about insurance than about cars, so when providing car insurance it may decide to partner with other organizations that know more about cars than about insurance. This results in a composition of insurance-related services and car-related services (for example, determining the insurable value of a used car). Such compositions can be either directed (in other words, composed by a single dominant player) or collaborative (in other words, emerging from the interaction between multiple players within the ecosystem).

For example, an online retailer knows about her products and services, but doesn't wish to become an expert in credit card handling, or to be responsible for data protection and security, so she delegates these concerns to a specialist provider that knows more about these matters.

Similar considerations can apply to industry consortia, such as ACORD (for the insurance market). ACORD can define generic service-based assets (such as models, schemas, interfaces and so on) for insurance. However, insurance companies will also wish to use generic service-based assets to cover requirements that are not insurance-specific, such as customer management or complaints handling, where ACORD may not be able to add any knowledge-value, and it would be appropriate for ACORD to regard these as peripheral to its own activities rather than core.

So one approach to Ecosystem SOA is to push out from the enterprise into the ecosystem. John Hagel calls this Inside-Out Architecture, which he contrasts with Outside-In Architecture. (See my post on Outside-In Architecture.)

An Outside-In Architecture starts with a model of (the flows of) knowledge and value in the ecosystem as a whole. The strategic question for an enterprise is how to find way of both contributing value to the ecosystem, and drawing value from the ecosystem, through the provision of ecologically viable services.

For example, a telecoms company might reasonably consider that its core competence is something to do with communications. So a positional strategy would drive it to a dominant position in the middle of a large communication ecosystem, providing a platform of services that add value to a diverse range of communication activities by other people. The dominant position would allow it to negotiate a strong share of the value generated.

However, respect for the ecosystem would lead it to leave sufficient value to third parties to maintain the economic health of the remainder of the ecosystem. Instead of a simple positional strategy, a relational strategy (based on mutual trust with ecosystem partners) should produce a more sustainable and ecologically sound ecosystem.

Is Enterprise Architecture a Profession?

2009-10-27T11:07:00.003Z

The Center for Advancement of Enterprise Architecture Profession (CAEAP) is in the process of developing an EA Professional Practice Guide that, among other things, will define what it mean to be an Enterprise Architect. Is Enterprise Architecture a profession, or does it have any reasonable chance of becoming a profession in the foreseeable future?

While I don't fully agree with the Wikipedia definition of Profession, I think it identifies a lot of characteristics commonly associated with professional status, and EA lacks many of these characteristics.

Here are three reasons why I don't think EA is (yet) a profession.

1. Organizations claiming to represent enterprise architects have relatively few members.

2. There are no meaningful sanctions against maverick or incompetent practitioners.

3. Although some universities are starting to offer degrees in Enterprise Architecture, for the foreseeable future this qualification will only be held by a tiny fraction of practitioners, mostly at the inexperienced end.
There are some worthy goals on the CAEAP homepage, which I think accord with my view of what counts as a profession, but these are currently merely future aspirations. I acknowledge that CAEAP seems to be putting a fair amount of effort into this, but this doesn't justify some of its more extreme supporters trying to confuse AS-IS with TO-BE. I shall be prepared to regard EA as a profession and the CAEAP as a legitimate representative body if and when these goals are ever achieved. For the time being, I think it is more accurate to regard EA as an aspiring profession than an established one.

However, the present interest in CAEAP is almost entirely coming from enterprise architects themselves, rather than from the people they are providing services to, and I think this may be a major obstacle in the CAEAP's achieving its goals.

Another major issue, in my view, is that of professional accountability. In healthcare, professional and ethical responsibility is clearly focused on the patient (as the "consumer" of healthcare services). In the legal profession, a lawyer represents a specific client and there are clear conflict-of-interest rules preventing a lawyer representing multiple parties in the same case. The CAEAP Value Map recognizes the need for "professional accountability" as well as "responsibility to multiple stakeholders and seek(ing) balance among potentially conflicting demands". But to which of these multiple stakeholders is the Enterprise Architect ultimately accountable, and who has the ultimate authority to resolve conflict?

I believe that CAEAP should be talking not just to EA practitioners but also to the consumers of EA services - whoever they might be - and I don't see any sign that they are doing this. (However, they are happy to accept sponsorship from vendors, who clearly have a commercial interest in influencing enterprise architects. This kind of sponsorship is perfectly acceptable for a trade body, but raises questions for an organization that wishes to establish itself as a proper profession. The Royal College of Midwives doesn't take backhanders from drug companies, does it?)

CLARIFICATION: Microsoft and IBM sponsored the CAEAP summit. @jpmorgenthal insists that this is a conference, distinct from the organization itself, and argues that the organization itself is not sponsored by vendors. That may be strictly true, but it is difficult to avoid the impression that sponsoring the conference significantly benefits the organization.

Jon Ayre challenges this point. "Consumers choose whether or not to consume. If EA doesn't provide right services consumer will reject it."

Of course, with some exceptions, professionals cannot force their services upon their clients. But there is still some external validation - a practice becomes a profession not because people inside the practice want it to be, but because people outside the profession have respect for it. Members of a profession may receive public or institutional funding for some of their activities, and may have a privileged status in legislation and regulation.

As far as I can see, all of CAEAP's directors and trustees are EA insiders. So where is the external voice? Who represents the stakeholders of EA?

Is Enterprise Architecture Dead?

2009-10-27T09:21:00.000Z

I can't see much recognition or respect for the value of "Enterprise Architecture" except among the ranks of EA practitioners. In many organizations, the function of enterprise architecture is squeezed or marginalized, if not rejected altogether.

The attempt by the CAEAP to turn EA into a "Profession" is not going to address this problem. The desire for professional status is not coming from the demand-side (CEOs wishing to distinguish genuine practitioners from charlatans) but from the supply-side (like teenagers wanting to be taken seriously). People who think EA is a waste of space are not going to be reassured by the existence of a cartel of people with impressive-looking certificates.

Meanwhile the most experienced and able practitioners are getting on with the work, engaging with the business rather than worrying about preserving a label with such negative connotations.

I have been mulling on this post for a while, but I am now prompted to complete it by CBDI boss David Sprott, who has just produced a good post on The Death of Enterprise Architecture. Perhaps responding to the fuss about the Death of SOA, which exercised a few minds earlier in the year (see my post Has SOA Gone for a Burton?), he suggests that it is Enterprise Architecture that is dead - not just in need of a new label but in need of a new concept.

David proposes Smart Ecosystem Architecture. He refers to some of the CBDI reports I wrote about ecosystems in various industries (Airlines, Insurance, Public Sector) and argues that it's not about the enterprise any more.

"Various influences particularly Complex Event Driven Architecture and Smart Business and IT are strongly predicated on optimizing business design and processes involving all the ecosystem stakeholders."

Not just engaging with the business, then, but looking beyond the business into the demand environment. See my paper (with Philip Boxer) Taking Governance to the Edge (Microsoft Architecture Journal, August 2006)

Event-Driven Enterprise Architecture

2009-10-26T10:35:00.000Z

@toddbiske responded to @jeffrschneider 's question about the trigger for EA services.

Todd distinguishes between two kinds of service, which he calls request/response-style and event-driven services.

"There are services that are explicitly invoked at the request of a consumer, and then there are services that are executed in response to some event. In the latter case, the team providing the service is the one monitoring for the event. If some other team was monitoring for it, and then told your team to do something, then we’re back to the request/response style with one team acting as consumer and the other acting as the provider."

But surely a request is an event? The EA team may have many conflicting demands on its time and attention, and so there is always a decision (perhaps driven by some policy) whether and how quickly to respond to a given request.

A kind of event that might trigger EA services is one that looks like "we think we might have a problem in such-and-such area". (This is what Philip Boxer calls a P-type service - rcKP - services at the edge.)

In this example, "we" could be the EA team or the EA customer or both together. The service might include clarifying whether there really is a problem at all, not just solving a known problem.

The event triggering this kind of work is typically a weak signal. There are usually more possible problems than we actually have time to work on, so the existence of a problem is not a sufficient trigger for activity.

One interesting question here is how the event-driven EA actually works. The EA team must have some view of a complex and dynamic world to which it is attempting to add value, that allows it to organize its work. In other words, it needs a model (possibly implicit, but preferably explicit). But what model is this? Not just the enterprise model itself, not just the enterprise architecture framework (TOGAF or what have you) but something else as well.

Towards an Architecture of Privacy

2009-10-25T13:41:00.000Z

@futureidentity (Robin Wilton) posted some interesting ideas about Identity versus attributes on his blog.

"For an awful lot of service access decisions, it's not actually important to know who the service requester is - it's usually just important to know some particular thing about them. Here are a couple of examples:

If someone wants to buy a drink in a bar, it's not important who they are, what's important is whether they are of legal age;

If someone needs a blood transfusion, it's more important to know their blood type than their identity."

However, there is an important difference between Robin's two examples. Blood transfusion is a transaction with longer-lasting consequences. If a batch of blood is contaminated, there seems to be a legitimate requirement to trace forwards (who received this blood) and backwards (who donated this blood), in order to limit the consequences of this contamination event and to prevent further occurrences.

There is a strong demand for increasing traceability. In manufacturing, we want to trace every manufactured item to a specific batch, and associate each batch with specific raw materials and employees. In food production, we want to trace every portion back to the farm, so that salmonella outbreaks can be blamed on the farmer. See Information Sharing and Joined-Up Services 1, 2.

Transactions that were previously regarded as isolated ones are now increasingly joined-up. The eggs that go into the custard tart you buy in the works canteen used to be anonymous, but in future they won't be. See Labelling as Service 1, 2.

Of course, there is also a strong demand for increased auditability. So it is not enough for the barman to check the drinker's age, the barman must keep a permanent record of having diligently carried out the check. It is apparently not enough for the hotel or bank clerk to look at my passport, they must retain a photocopy of my passport in order to remove any suspicion of collusion. (The bank not only mistrusts its customers, it also mistrusts its employees.)

There is a large (and growing) class of situations where so-called joined-up-thinking seems to require the negation of privacy. I am certainly not saying that this reasoning should always trump the needs of privacy. But privacy campaigners need to understand that all transactions belong within some system of systems, and that this provides the context for the forces they are battling against, rather than pretending that transactions can be regarded as purely isolated events. The point is that authorization is not an isolated event, but is embedded in a larger system, and it is this larger system that apparently requires greater disclosure and retention.

@j4ngis asks how long chains to use for traceability. What "length" of traceability is sound and meaningful? How do we connect all these traces? And also backward and forward in the "chain". For how long should records be kept?

Should we also know the batch number for the food that was given to the chicken that laid the egg you included in the cake?
Do we have to know the identity of the blood donour after six months? 10 years? 100 years?

The trouble is that there is no rational basis for drawing the line. It is always possible that some contamination in the chicken feed might affect the eggs and thereby the custard tart. It is always possible that the hyperactivity of certain schoolchildren, or the testosterone levels of certain adults, might be traced back to some contamination in the food chain. It is always possible that some obscure data correlation might one day save lives or protect children. And given the vanishing costs of data management, even a faint possibility of future benefit appears to provide sufficient reason for collecting and storing the data.

Robin clearly supposes that attribute-based authorization is a "Good Thing". I am sympathetic to this view, but I don't know how this view can stand up against the kind of sustained attack from a certain flavour of joined-up systems thinking that can almost always postulate the possibility (however faint) of saving lives or protecting children or catching criminals, if only we can retain everything and trace everything.

For my part, I have a vague desire for anonymity and privacy, a vague sense of the harm that might come to me as a result of breaches to my privacy, and a surge of annoyance when I am required to provide all sorts of personal data for what I see as unreasonable purposes, but I cannot base an architecture on any of these feelings.

Traditional arguments for data protection may seem to be merely rearguard resistance to integrated and joined-up systems. Traditional architectures for data protection look increasingly obsolete. But what alternatives are there?

Asymmetric Demand for Defence Equipment

2009-10-15T22:21:00.060+01:00

An independent review into the way the MOD buys equipment for Britain's Armed Forces has been published today, Thursday 15 October 2009. [Report, MoD News Article, BBC News]. Key finding.

"The Ministry of Defence has a substantially overheated equipment programme, with too many types of equipment being ordered for too large a range of tasks at too high a specification. This programme is unaffordable on any likely projection of future budgets."

That situation might sound familiar to a lot of managers, not just in the defence sector.

The report makes some favourable comments about the Through Life Capability Management (TLCM) programme, but indicates a lack of hard financial data that would be required to make quantitative decisions. There has been some discussion along these lines published in the RUSI Journal, including Agility and Innovation in Acquistion (Feb 2008) and The Meaning of Value-for-Money (Feb 2009).

The explanation for the current crisis can be found in the essential multi-sidedness of the defence acquisition ecosystem. Traditional cost accounting approaches (such as activity-based costing) fail to address the complexity of this multi-sidedness, and researchers are urgently seeking alternative cost accounting methods appropriate for complex systems-of-systems.

One of the key issues for Through Life Capability Management is that any errors or omissions in the long-term equipment programme must be repaired through what are known as Urgent Operational Requirements (UOR), which over the long haul can prove far more expensive and inflexible than the planned equipment.

The report also praises the Smart Acquisition programme, and expresses regret that the disciplines of Smart Acquisition have been somewhat diluted by recent reorganization.

Is this report only relevant to the defence sector, or can other sectors glean anything useful? My view is that the complexities of multi-sided markets and asymmetric demand can be found in many, perhaps most sectors. And the question of coordinating effectively between short-term and longer-term spending can be found in many domains, notably IT. I have little doubt that whatever management tools and techniques are developed by the MoD and its partners to address this problem will eventually trickle into civilian management.

Online pricing practices to be regulated?

2009-10-15T20:51:00.000+01:00

The British Office of Fair Trading (OFT) is starting an investigation of online pricing practices. [BBC News, 15 October 2009]

Type	Description	Typical Offenders
Drip pricing	Consumers only see an element of the price upfront and end up paying much more due to optional or compulsory extras	Airlines, car hire firms and insurance companies
Time-limited offers	For example, sales that finish at the end of the month or last for one day only.	Carpet stores and furniture sellers
Baiting sales	A company advertises discounts to attract visitors whilst having few items at that price on sale
Reference prices	Artificially inflating the pre-sale price of an item in order to make the discount look more attractive.	Companies offering cruises or selling furniture. Supermarkets
Complex pricing	It is difficult for a consumer to assess an individual price, such as with three-for-two offers and 'free' add-ons.	Mobile phone companies, supermarkets and computer stores
Customized pricing	Prices are individually tailored using information collected about a consumer's internet use
Price comparison websites	There may be some hidden financial ties or other collusion between an apparently independent comparison site and the suppliers whose prices they are comparing

I have talked about some of these pricing schemes and scams before, in particular complexity-based pricing. I've also talked about ways (such as the infamous Ghetto Latte) whereby consumers can get a better deal than the service provider intended. All this kind of thing results in chronic distrust between service provider and service user, and excessive transaction costs. Might seem like a classic argument for regulatory intervention, if we could really believe that would make the situation fairer. What do you think?

Disruptive business model for telephony

2009-10-09T01:28:00.000+01:00

@martingeddes believes that "voice telephony is a market full of potential for the service providers, applications developers and innovators, and full of promise for both end customers and upstream business organisations such as contact centre operations" (New business model aims to bring voice back, BT plc Innovation).

"What would really be disruptive is the complete turning upside down of the telephony business model."

Martin identifies three key challenges, which roughly correspond to the three asymmetries identified by Philip Boxer and myself in Metropolis and SOA Governance (Microsoft Architecture Journal, July 2005).

Martin's first challenge is how users connect. “In the case of contact centres, traditional outbound calling can be unproductive for agents with an average of around fifty per cent of an agent’s time being spent on useful calls.”	The third asymmetry requires separating out the different contexts of use
Martin's second challenge is how upstream business and end-users interact. “This is where we would need to integrate our understanding of the customer with what the upstream organisation wants to do to interact with them. Small, network and applications-based intelligent telephony improvements that enable this to happen could lead to profitable interactions for businesses and better experiences for end-customers.”	The second asymmetry requires separating out business models that can organize supply from the solutions that are on offer.
Martin’s final challenge relates to how transactions will work. “The key here is developing and implementing intelligent telephony services.”	The first asymmetry involves separating out technology from the supply of specific products.

To become better at capturing asymmetric forms of demand, an organization such as BT needs leadership that will enable it to do two things:

Take power to the edge of the organization: The people at the edge of the organization with the relationship to the asymmetric demand must be able to organize the business model they need to capture that demand.

Develop an agile infrastructure: providing business services that can be orchestrated and composed at the edge in response to the particular forms of demand they are targeting. This then allows the supply-side of a business to extract economies of scale or scope when providing support across multiple business models.

Is Architecture the Enemy of Innovation?

2009-09-21T12:55:00.000+01:00

asks @EnterprisingA . Well, obviously architecture is not the avowed enemy of innovation. As Jon points out, they ought to be on the same side, because there are some strong shared values.

But surely the real question is not whether they ought to be helping each other, but whether (good intentions notwithstanding) they actually do support each other's efforts more than they (inadvertently) hinder and interfere with each other.

I think there is some evidence for the proposition that Architecture and Innovation don't actually collaborate as well in practice as they ought to in theory. This does not necessarily mean a critique of Architecture as such, but a critique of current "best practices".

Jon talks about the TO-BE architecture, but I am also interested in the TO-BE practice of architecture.

Economics of agility 2

2009-09-02T14:46:00.000+01:00

In my previous post on the Economics of Agility, I noted how little material has been published on this topic.

As Nicholas Whittall and Philip Boxer point out in their contribution to the recent debate on The Meaning of Value-for-Money in Defence Acquisition (RUSI, February 2009), there is an important link between agility and alignment. See also their earlier piece on Agility and Innovation in Acquisition (RUSI, February 2008).

The first observation is that defence acquisition - just like systems acquisition most anywhere - operates on a much slower tempo than the requirements of the business. The "business" of a military organization is running military campaigns; thus when writing for the defence community, Whittall and Boxer refer to the Campaign Tempo and the Acquisition Tempo.

The second observation is that there is a complex set of activities (such as orchestration, customization, and improvisation) involved in bridging between Demand (the demands of the campaign or business) and Supply (the procurement of specific systems and devices). These activities operate on an intermediate tempo, which Whittall and Boxer call the Alignment Tempo.

"Meeting the campaign tempo then depends on the alignment tempo possible, which in turn depends on the acquisition tempo at which gaps can be filled. Any slowness in acquisition tempo leads to increased bricolage and process short cuts to enable the alignment tempo to keep up with the campaign tempo. Thus, ‘agility’ finds its richest expression in the ability of the alignment tempo to meet the required campaign tempo at the lowest cost – i.e. to maximise the value-for-defence."

The challenge is then to produce just enough variety within the acquisition to optimize the economics of alignment. Boxer has developed a technique of Cohesion-Based Costing (not yet published), which "offers a means to attach a value to the cost of introducing flexibility". This kind of technique will clearly be of enormous benefit within the SOA world.