soabloke

An Hypothesis

2010-02-16T07:27:35Z

Enterprise IT is badly broken in most of it’s current incarnations. This is due to a wide range of influences – organizational, political and technical. I think there are ways to increase productivity and reduce cost and risk by changing enterprise IT practices towards:

A collaborative business and IT culture that owns and manages risk rather than just outsource and lose control.
An Enterprise IT strategy and Architecture that provides “cohesion” so that smaller, agile projects can deliver strategic value.
Using (and developing in-house) development frameworks that enhance productivity by:
- encouraging in-house standards and best practices,
- operating at a high level of abstraction – business processes, services, events, rules
- foster developer skills beyond “commodity” levels.

The Year of Living Asynchronously

2010-01-11T12:22:27Z

Happy New Year! Asynchronicity is busting out all over the web and my prediction is that 2010 will be the year of “events”:

Of course Twitter has brought The concept of publish/subscribe messaging to the masses and we enjoyed their journey of discovery to the heights of scalability in 2009.
XMPP has been embraced by the real-time web crowd, most publicly in Google Wave but also in other “back-web” contexts such as Gnip.
Web sockets is an experimental feature of HTML 5 which enables push messages directly to web pages.
New frameworks for event-driven programming are emerging such as EventMachine, Twisted, Node.js.
In 2009 every major software vendor had a CEP product.

In the meantime, SOA has become so damn synchronous. But it doesn’t have to be!

One of the fundamental tennets of SOA is that reducing coupling between systems makes them more scalable, reliable and agile (easier to change). SOA goes a long way to reducing coupling by providing a contract-based, platform independent mechanism for service providers and consumers to cooperate. However I still think we can improve on current SOA practices in further reducing coupling.

Coupling still intrudes into many aspects of how SOA is practiced today:

HTTP transports tie us to a regimen of synchronous request-reply with timeouts which creates tight couplings between provider and consumer. Even though one-way MEPs were an original feature of SOAP, message-oriented transports remain the forgotten orphan of web-services standards.
Many SOA services are conceived, implemented and maintained as point-to-point entities…providers and consumers forced into lock-step due to inadequate versioning and lifecycle management.
Process orchestration layers often form a bridge between service providers and consumers, which on the face of it provides some level of indirection. But in many cases orchestration provides limited value and may actually serve to increase the overall system coupling.

In many cases we can achieve the benefits of service orientation to much greater effect by exercising a little scepticism toward some of these shibboleths of the web services world and embracing a more asynchronous, event-oriented way of building processes. So this year, embrace your asynchronous side and do something to reduce your system coupling: build some pub/sub services, learn about Event Processing or Event-Driven Architecture, try one of the technologies I pointed to above.

Just as developers should embrace multiple languages to broaden their skills, so should architects embrace and be fluent in multiple architectural styles.

Reductio ad Lucidus

2009-11-11T00:51:15Z

In a recent comment on my Architectural Characteristics posts, Andy astutely observes that I may be “shoe-horning”. By this I assume he means that I’m taking a large and rather lumpy concept and trying to squeeze it into a smaller and more uniformly shaped container while risking some distortion in the process. I’ll admit that in this respect I’m probably guilty as charged.

But I should clarify my purpose in doing this. I’m trying to cut back the various architectural styles under consideration to a simpler form where the essential characteristics can be discerned without confusion from some other non-essential characteristics. So rather than shoehorning, I’m trying to setup a strawman model which can be used as a starting point for discussion. Or maybe like a physicist I’m trying to model a very complex phenomenon using linear approximations which explain the broad outlines of the phenomenon at the risk of falling short on some of the details.

To extend this latter metaphor, I don’t think it is too much of a stretch to say that the architectural styles I’m considering could be likened to “fundamental” architectural styles and that real-world architectures could be viewed as “superpositions” of those fundamental architectures.

If we consider the simplified forms of EAI and SOA that I describe, each style falls short of representing a real world architecture, but the upside is that the EAI and SOA styles as I describe them are distinct and easily differentiated. So we have a model which provides a way of distinguishing between different styles (via the characteristics I’ve discussed) but falls short of exactly matching a “real-world” architecure.

If we look at any real-world architecture in recent years, I think we can see a superposition of EAI and SOA concepts. This probably reflects an evolutionary path between the two styles. EAI as practiced in the early noughties had already developed the idea of a normalised data model and technology independent interfaces. These were not standardized, but some of the characteristics of SOA were apparent in what was then called EAI.

Similarly, EAI was not always about data integration. There was (and is) a distinction between data integration and process integration. EAI techniques could be used to orchestrate processes across multiple systems. This is even closer to the concept of SOA which has at its core the notion of an independent process layer seperate from the service layer.

Even if we don’t superpose EAI and SOA into one solution, there are still legitimate ways in which EAI, SOA and EDA coexist within any particular architecture. We can easily imagine a solution in which a business process is orchestrated via SOA services, reference data is synchronised using EAI and overall process state is monitored using EDA techniques such as Event Processing.

So real-world solution architectures exhibit some overlap between the different architectural styles – EAI, SOA and EDA. Some of this is due to evolutionary legacies, or due to plain-old confusion between the different styles (e.g. JABOWS as really being EAI). Some of it is also due to legitimate mixing of different styles for different aspects of a solution.

I think that real-world architectures can benefit from seperating out the “essence” of each architectural style and being explicit about how those styles are being applied. Reducing architectural styles to simplified forms clarifies the stucture of a real-world architecture. Not very different from Design Patterns, really.

The SOA Manifesto

2009-10-26T22:20:16Z

Taking a leaf from the Agile playbook, a group of SOA thought leaders has put together the SOA Manifesto, a succinct list of SOA principles and preferences to guide Service Oriented Architecture. Great work!

(I could comment further, but it speaks for itself).

Go visit and find out.

No related posts.

Characterising Architectural Styles II – State

2009-10-26T00:44:10Z

My last post explored some distinguishing characteristics of three common architectural styles in an attempt to understand better how they differ and therefore how they may apply in different contexts. A fourth distinguishing characteristic of these architectural styles is the way in which state is managed during the execution of a business process.

There are three aspects of state that I want to consider:

Management - how state change is initiated or managed through a business process.
Monitoring - how state is monitored or accessed or derived during the execution of a business process.
Consistency - how state is made consistent across different systems involved in a business process.

A summary of the state characteristics of the different architectural styles is listed in the following table along with the other characteristics discussed in my last post.

In EAI, state is managed within one application and then synchronised to other applications, usually after the business process has completed. This means that in some cases state may be inconsistent across the organisation or its systems. That may or may not be a problem (see ‘Eventually Consistent‘) but is a common side effect when a business process is executed within one system rather than across systems in an independent process layer. During the execution of an EAI process, there is often no monitoring of the state. I.e. the process may simply replicate data changes to other systems without explicily tracking state. A limited form of state monitoring may exist in the sense that the local application or associated middleware may check the status of data synchronization and error out in the event of an exception. I refer to this as ‘local’ state monitoring. So under the EAI architectural style, state is managed locally, monitored locally (if at all) and is eventually consistent.

Under SOA, the driver of a business process is BPM orchestration in an independent process layer. In this case, we can say that state is managed centrally (in the BPM layer) and monitored centrally (also in the BPM layer). State in end-systems is updated progressively through the business process (via service calls) and so we could say that state is ‘progressively’ consistent as opposed to ‘eventually’ consistent.

Under EDA, there is no central or even local management of state. Instead, events signify distributed actions which together may be used to infer the state of a system. To the extent that any ‘management’ occurs, we could say that state is managed in a distributed fashion – one or more agents each acting on their own. Perhaps it is more accurate to say that state is manifested globally. Converesely, state is monitored centrally within an Event Processing (CEP) layer which correlates events to infer system state. Under EDA, state is progressively consistent because the system is progressively reacting to events which are a by-product of a hidden or implicit business process.

Characterising Architectural Styles

2009-10-14T06:04:29Z

I’ve recently been thinking about simple ways to characterise the different architectural approaches we use in distributed systems today. Three simple architectural characteristics I have come up with are:

Asset - this is a core capability or “thing” that must be built, procured, maintained and managed in a corporate IT “inventory”.
Element - these are the “atomic” building blocks used in the process of Composition.
Composition - this is the mechanism which allows the different assets to work together to support a business requirement.

If we apply these characteristics to three common architectural approaches then we get the results in the following table.

EAI, although unfashionable is still prevalent – even dominant in the industry. For EAI, we are primarily concerned with applications (usually COTS) which embody and support the requirements of different parts of the business. Multiple applications must be coordinated to support the whole business. The primary coordination mechanism under EAI is synchronization of state between the different applications – primarily via data integration. The composition element is the API.

SOA is characterised by Services as the key asset. Services are acquired or built to execute business operations. Elemental Services are composed to support business processes – a sequence of operations which results in a business outcome. BPM (or process orchestration) is the composition mechanism within SOA. The underlying functionality of a Service may reside in one or more applications, but from an SOA perspective this is of secondary importance…SOA is concerned with the Service, not necessarily its implementation.

EDA (Event Driven Architecture) is characterised by Event Services as the key asset which represent an asynchronous notification of an important event associated with the business. Elemental Events are correlated and further processed to derive higher-order business intelligence which may in turn trigger other Events. The primary composition mechanism within EDA is Event Processing (or CEP). An important part of this composition mechanism is the ability to manage or track system state.

This characterization gives more clarity to the difference between JABOWS and true SOA. Many so-called SOA projects have simply involved the “bottom up” exposure of application APIs using web-services standards – resulting in “Just a Bunch of Web Services” which don’t realise the business value of a true SOA. JABOWS is EAI because the applications are the core “asset”. A true SOA has a “top down” process-centric architecture with Services as the core asset.

Characterising Architectural Styles II – State

The Science Debate is Over – Now We Need An Effective Response

2009-10-04T22:38:37Z

Tony Abbott says that he is unconvinced by the science behind climate change. I’m guessing that like most of our politicians, Mr Abbott has no science education beyond High School, so he is making this science value-judgement from a position of ignorance.

Most politicians would not provide financial advice, or medical advice, or advice on how to build a skyscraper because they know it is irresponsible and dangerous to voice an uneducated opinion on complex and technical matters. So why do politicians like Mr Abbott and Mr Fielding persist in taking a stance on climate science which is in direct opposition to the consensus of the world’s leading scientists?

What would convince Mr Abbott? Is he gathering new data on molecular resonances in the upper atmosphere? Is he running new oceanographic models on his parliamentary PC? No, Mr Abbott simply chooses to deny the problem because it absolves him from
taking any action. As a politician, Mr Abbott is not qualified to make value judgements on the science. The overwhelming,
peer-reviewed scientific consensus is that man-made climate change is a clear and present threat to the natural systems
that support humanity. As a politician, Mr Abbott should be working on policy and legislation to address the threat from
climate change.

If Mr Abbott doesn’t like the proposed Emissions Trading Scheme (ETS) then there are a number of valid politicial responses he can make. A valid political position might be that the proposed ETS is not the best way to control carbon emissions – but then he would need to come up with a better alternative. Another valid political position might be that the short term economic costs of the proposed ETS are too high – but then he would have to address the long-term economic costs of doing nothing. Simply denying the problem is not a valid political position.

Politicians should not be debating climate science – they have nothing new or valid to add. They should be taking immediate action to address the threat from climate change. We need effective policy and most of all we need leadership.

</rant>

Climate Change Update

Emergent Architecture

2009-08-19T00:25:12Z

Dion Hinchcliffe has written a nice blog entry “pragmatic new models for enterprise architecture take shape“. This resonates well with some of the things I’ve been saying about how enterprise architecture needs to be enabling rather than blocking:

“In recent years enterprise architecture has been moving from a discipline that provides top-down, a priori technology blueprints to the business side to one that articulates key, strategic possibilities and only the most critical high-level constraints (such as security standards) and then operates as a conductor, promoter, problem solver, and evangelist across the organization through the vehicle of a cohesive community to co-develop needed solutions.”

(my emphasis added). And:

“Invariably, the best architecture I see comes naturally from self-organizing thought leaders in an organization that seek each other out and collaborate on common solutions to their problems. Rather than the us vs. them mentality of old-world enterprise architecture, there is only an us mentality. Instead of prescribed standards, designs, technologies, and tools there is real [two-way] consensus and immediate buy-in.”

And as with all Dion’s posts there is the cool graphic. I especially like the two downward facing arrows – “community leadership” and “guides”:

The article is well worth reading but I think there is a large element of wishful thinking here. It paints a good picture of how enterprise architecture should be, but I fear we are a long way from this nirvana. Much of the vision depends on a fundamental change in organizational culture. Businesses that recognise the value of an “emergent culture” will naturally have an “emergent enterprise architecture.” I don’t think it will go the other way around. Emergent enterprise architecture will not survive in or be able to change a top-down corporation.

In the long term, those companies with the right culture will support emergent enterprise architecture and thrive on IT success. Those companies that don’t will cede their IT resources to others. Those who can do IT will do it on behalf of those who can’t. Cloud anyone?

EA and Agile Projects

2009-08-14T09:33:17Z

A recent comment on my post about the value of enterprise architecture suggested that enterprise architects should be directly involved in Scrum projects. In that post I suggested that the primary value of EA is “continuity” and “self-correction” across all the IT initiatives. If these qualities represent the value of EA, then the mechanism for realising that value must involve communication.

Communication is the key to ensuring that IT project deliverables provide cohesion across organizational, geographical and temporal boundaries. I have previously written that communication and governance are effectively the same thing. The problem is that “governance” needs to overcome the image of heavy-handed, top-down, hierarchical “rules” handed down by “out of touch” enterprise architects – like commandments on tablets of stone. EA must be seen as an enabler, not a roadblock.

But communication and governance must also be relevant to the organizational culture. So if the culture embraces agile development methods such as Scrum, then on the face of it the governance structure ought to conform with agile practices which favour direct verbal communication over documentation. If it is true that an embedded business representative can communicate requirements more effectively than a written requirements document, then perhaps it is also true that an embedded enterprise architect can “govern” more effectively than a written EA document.

However, on reflection, I don’t think the analogy of EA as a “customer” of the project is quite as strong as I first thought. There are a couple of differences between “business requirements” and “EA requirements”.

First, business requirements for a project are communicated primarily to that project. Few if any other IT projects would be interested in these particular business requirements which we are trying to encode into an IT solution. However, EA requirements must be communicated consistently across multiple projects and consistency demands that the EA requirements – the “governance principals” – need to be encoded in some written form, independent of an individual architect or even team of architects.

Second, business requirements change very frequently within the timespan of an IT project. Indeed the development process itself can change business requirements. This is why agile methods value verbal communication methods over written – so as to allow for frequent change in business requirements. EA requirements will not change as rapidly. Even though the Enterprise Architecture will evolve over time, it should not change over the course of a single project. Indeed if your Enterprise Architecture undergoes sudden rapid shifts (e.g. adopting an overnight SOA mandate) then you have a recipe for IT disaster. Hence the agile “insight” that written business requirements cannot keep up with reality, is not quite as relevant when it comes to EA requirements.

So my response to Paul’s comment is still: Yes! I agree that an EA representation on a Scrum project would be a good idea – to communicate, elucidate and interpret EA requirements to the project and importantly to take new lessons learned back from the project into the governance structures. However, this is no substitute for EA “documentation” – artefact-based governance – which is necessary to ensure consistency and continuity.

Health Memes Again

2009-07-24T12:35:33Z

One of my stints in the US was in the early nineties when I lived in Baltimore MD, at that time the shootingest city in the Union. My daily (bus) commute from South Baltimore across North Avenue to Johns Hopkins University was an eye-opening tour across the strata of American society. First Lady Hillary Clinton was pushing her pet project for universal healthcare in the US, which actually gained quite a bit of traction and certainly got the whole country talking. NPR ran endless reports comparing US health standards with European. But then some Republican stood up and said “you can’t have universal healthcare, that’s…Socialism!“. The public response was unequivocal:

“Oh yeah, we forgot. Sorry…”

Twenty years later, the USSA seems to be waking from its long healthcare slumber and taking some action – perhaps emboldened by having recently nationalised all the banks. This time, Information Technology seems to have a strong role to play as the discussion encompasses electronic medical records (EMR) and the part that IT can play in making healthcare work more efficiently.

Clearly this is a lofty and worthy goal for IT, but there are also some really interesting technical aspects to the application of IT to EMR. There are the hard-nosed “enterprise” requirements for scalability, reliability and security across a complex and distributed system-of-systems. Add to this our experiences from “Social Software” and the role it can play in helping different parties – physicians and patients – collaborate on maintaining accurate medical records. Knowledge Management, Semantic Web and Decision Systems also have a contribution in helping to automate decisions and discovery in a highly technical, specialised and always-changing field. EMR has it all!

So it’s not surprising to see a lot of discussion on this topic in the blogosphere. A few interesting references recently:

First welcome back Adam Bosworth with a series of great posts on EHR to launch his latest venture. I missed Adam’s writings since 2005 when he disappeared into the Googleplex. Adam bolts out of the gate with ten posts in one day (I suspect he just discovered his network cable unplugged for the last three weeks).

John Udell presents an interesting podcast with Peter O’Toole discussing Electronic Medical Records and (among other things) the application of expert systems to aid the entry of specialised data.

Finally (but not least and I think not last), Robert Cringley starts a series on the application of IT and complex systems theory to medical records.

Certainly an interesting area to watch with ramifications not only in the US, but certainly in Australia and across much of the western world where healthcare reform and efficiency is firmly on the agenda.

No related posts.