I'd be the first one to recognize that I am playing way out of my league here, no question about it, but it seems to me that Computer Science has been neglecting, if not abusing, the notion of state in ways that seem to be counter productive. I understand that the work of Church and Turing was all about "Computing" and not surprisingly when we started to look for better abstractions to write programs the foundation of λ-calculus was never reassessed but it is difficult today to not ask the question as to whether Abstract Data Types are at the foundation of the General Theory of Software Engineering (GTSE).

I spent the last couple of weeks pounding on the idea that there is more to Barbara Liskov statement about the fundamental role of programs than Abstract Data Types: 

I believe that the fundamental role of programs is to modify state not just the bits

Indeed, when you look at it closely, the role of "computers" today is far less about "computing" than it is about managing state, even though computing is required to transition from state to state properly.

After defining BOLT, and looking at extensions in the problem domain, I found that the idea of defining a programming model based on Abstract State Types could shed some new light on what a programming language truly is.

Surprisingly, there is very little literature about ASTs as I define them. It seems that ASTs fit the model of Coalgebras, but again, I am playing way out of my league here. 

In the paper referenced below, I define an AST as a simple data structure (key value pairs) which all share the same lifecycle (yes, states have a lifecycle too!). A state lifecycle is triggered by an interaction between two (or more) states. States are immutable and new states (of the same type or a different type) are created at the end of an interaction. I define 3 composition mechanisms (containment, extension, connection). Interestingly, there is no ASM (Abstract State Machine) behind ASTs. What we see as a "State Machine" is actually the occurence of repeatable interactions, so all the complexity associated to modeling state machines goes away.

If anyone feels there is a strong analogy between ASTs and QED, well, it is nearly by design, but it really came about from me staring at UML State Diagrams, linking them to business process execution, then to business strategy and then all the way back to problem definition and today programming model. 

This paper is far from complete, a very rough draft at best. I publish it as is for three reasons:

a) I strongly believe that the fundamental idea behind ASTs will not change

b) It would take me an infinite amount of time, which I don't have, to bring it to maturation

c) This is really a call for help, if someone would like to jump in as a coauthor, to further the idea

I don't see ASTs as being the next gen programming concept, ADTs are just fine, but it was hard to resist not extending the BOLT conceptual framework to the programming model level, especially in the context of Barbara Liskov's statement on the role of programs.

Here is the paper, as usual any feedback is welcomed, especially if someone has already worked on that idea. My next step is really to put this idea in practice to explore it further. In particular the paper does not speak at all of the general orchestration behind state interactions (mechanisms such as addressing, activation, ...) but this is by design because I feel it is totally orthogonal.

Again, I welcome the contribution of one or two co-authors, otherwise I'll continue working on this paper in an open content mode.

I have an infinite respect of Prof. Liskov, for her, personally, and her work. A lot of what she has discovered has guided my own work and the work of millions more for the past four decades and her insight on the role of programs has litterally transformed and shaped the way I think:

I believe that the fundamental role of programs is to modify state not just the bits

But, what if her fundamental assumptions were off? After all they are assumptions and assumptions need to be revisited periodically, they are (somewhat) artificial, often unconscious, constraints that we believe make it easier to find the solution of a given problem. Just ask Richard Feynman.

After reading, and rereading her seminal paper on "Programming with Abstract Data Types", it is difficult not to question her assumptions. 

For centuries, we have created valuable abstractions that make it easier to solve problems, very complex problems. As we all understand the value of high level programming languages, it would be hard to disagree, for instance, with her introductory statement:

Unfortunately, it is very difficult for a designer to select in advance all the abstractions which the users of his language might need. If a language is to be used at all, it is likely to be used to solve problems which its designer did not envision, and for which the abstractions embedded in the language are not sufficient.

Yet, that statement creates an unfortunate bias that somehow the language designer and the solution developer must always be two different people. It also creates another unfortunate bias, as it, somehow, suggests that every a team of developers should use the same programming language, maybe within a tier, if not for the entire solution stack.

Actually, it is probably very difficult to not count these two assumptions as mere facts. My career started pretty nicely with the development of advanced semi-conductor deposition systems. I used NeXTStep to build these systems: ADTs, OOP, and Model-Driven development tools (Interface Builder) offered a delightful way to create a model of these systems. These abstractions were so powerful that I ended up building a Natural Language Processor on top of the control systems (using Objective-C's reflection mechanisms). Back in 1996, this control system was jaw dropping, but to be fair, NeXTStep, ADTs, i.e. abstractions, combined with some elbow finger grease was all that was needed to create that kind of magic. Retrospectivelly, I trully think this was magic, and I could have never build such a system by myself without all the good work that was done at that time. 

In 1997, when Steve Jobs returned to Apple and killed NeXT, I had to leave this wonderfully abstracted world and decided to join the ranks of Information Technology professionals. IT was very different from the world of industrial process control: none of these abstractions make any sense in IT. When I say any, I really mean any. Everywhere I turned, ADTs didn't help: BPM (process), SOA (service), XML (eXtensible Data Structures), HTTP (Protocols), SQL or NoSQL (Data).  Worse, ADTs seemed to make things far more complex than one would have originally thought.

So why not aiming at a completely different direction? 

• What if every development team would be allowed to build (and have control over) their programming language?
• What if every developer in that team would have the flexibility to create their own programming language, if it made sense? 
• What if these programming languages could even be solution specific, again if it made sense? 

I know, this is a radical idea. I just ask you to consider the following proposal. The work of a team of developers would look like something like this: 

Ultimately Metaprogramming delivers 3 key benefits:

1) As the project progresses, you end up mostly editing the solution model. Writing code at this level is usually very compact and very stable. This means that you can make major functional changes up to the last minute, if you need to

2) You can refactor the code generator fairly easily, for that app, the code generator is about 10k LOC, while the generated code is slightly above 100k LOC. As you refactor the technology, this is about 90% less effort

3) QA costs go way down, again as you are in effect stabilizing far less amounts of code

This post is a synthesis of two posts I originally published on my other blog "unRelated".

One of the key foundations and most attractive principle of Agile or Lean methodologies is that  "Everyone can help each other remain focused on the highest possible business value per unit of time".

I am certainly a strong supporter of that principle. However, value is often difficult to assess, I would actually argue that it is easier to identify what has less or little value, but what we think as valuable can potentially lead to many false positive, or simply be "business-as-usual" and hide the broader structure of the solution. 

"User Stories" are the corner stone of identifying and delivering value:

An argument can be made that the user story is the most important artifact in agile development, because it is the container that primarily carries the value stream to the user, and agile development is all about rapid value delivery.

In practice, very few people focus on the benefits part of a user story. All user stories I see are either what we used to call "requirements" (just phrased slightly differently but isomorphically) or "tasks" needed to advance the state of the project.

However, there is a fundamental flaw in the construction of user stories, even when they are properly written, because they somehow make an assumption about the shape of the solution, and drive the author to turn almost immediately in solution mode, leaving no room for creative and out-of-the-box thinking.

Let's compare the metamodel of a User Story and to the formal definition of a Problem. The metamodel of a User Story looks like that (using the BOLT notation):

As a <role> I want to <action> so that <benefit>

I define a problem formally as a non existing transition between two known states [1],  the metamodel of a problem looks like that:

A solution is a way to transition between these two states. Please note that both the actors and the actions are part of the solution:

This is where the problem lies when using User Stories, you are specifying the requirements with the solution in mind. There is, of course, a general relationship between some of the actors and entities of the system with the "start" and "end" states of the problem. The problem states are always defined in terms of their respective states (possibly as a composite state), but it is a mistake to think that the actors and entities that perform the actions, as part of the solution, are always the same as the actors and entities related to the (problem) states.

Hence, an action is solution centric and should not be part of the problem definition. As soon as you pick one, you have put a stake in the ground towards the direction you are going to take to solve the underlying problem. The other issue is that the start and end states are never clearly identified in a user story leading to confusion in the in the solutioning and verification process, since the problem is not defined with enough precision. Benefits could sometimes align with the target/desirable state, but the definition is often too fluffy and more goal centric, not effectively representing that (problem) state.

Ultimately, the relationship between problems and solutions is a graph (states, transitions as problems, actions as solutions), and this is where the coupling between the problem space and the solution space at the User Story level becomes unfortunate. This means that User stories cannot be effectively nested and clearly cannot fit in hierarchical structures (which is common to most Agile tools I know). This problem is quite accute as teams struggle to connect business level user stories and system level or solution level user stories. The concept of having a single parent directly conflicts with the possibility of having multiple possible transitions into a single state and decomposition principles where the same problem appears in the decomposition of several higher level problems. 

I feel that distinction is profound because we can now clearly articulate:

a) the problem statements with respect to each other (as a graph of states and transitions)

b) we can articulate the solution in relation to the problem statements

c) we can articulate the verification (BDD) in relation to the problem and solution [2]

d) we can actually articulate the Business Strategy [3], the Problem Statement, the Solution and the Verification with the same conceptual framework

e) derive the Organizational IQ from the problems being solved on an every day basis

To the best of my knowledge none of these articulations have been suggested before and no one has ever provided a unified framework that spans such a broad conceptual view from the Business Strategy to the Verification. In the proposed framework the business strategy is simply a higher level and specialized view of the problem and solution domains, but using the exact same semantics (which are described here). In other words the enterprise is a solution to a problem, which is a composition of smaller problems and more fine grained solutions, etc. This has an extremely important implication for the execution of the strategy because now both the Strategy and its Execution are perfectly aligned, at the semantic level: the strategy, problem, solution and verification graph represent a map that everyone in the organization can refer to. 

To take advantage of this new conceptual framework. I suggest that we make a very simple and easy change to Agile and replace "user stories" by "problem statements". Each problem must be "solutioned", either by decomposing it into simpler problems or solutioning it directly. Value can still be used to prioritize which problems are addressed first, that part of the Agile and Lean movement is very valuable, so too speak, but the focus on problems and solutions opens a new flexibility in how we handle the long range aspects of the solution while enabling the highest level of creativity and ultimately a direct articulation with the IQ of the organization. 

As problems are decomposed, we will eventually reach a point where the subproblems will be close to or isomorphically related to the solution. But it would be a mistake to not clearly delineate the problems from solutions, simply because at the lowest level, they appear isomorphic. 

If we start drawing some BOLT diagrams, a problem lifecycle can be defined as:

The fact that the lifecycle is pretty much identical as the one of a user story enables most of the Agile processes and tools to work nearly unchanged.

You may want to know "How do I write a Problem Statement?". Personally, I don't like canned approaches. Oviously here, the mere definition of the two states (low value and high value) is enough to describe the problem. If a solution already exists (i.e. it is possible to transition between these two states) you may want to describe some characteristics of the new solution. I googled "How to write a Problem Statement?" and I felt there was already a good alignment betweent the results and the abstract definition provided above. For instance:

We want all of our software releases to go to production seamlessly, without defects, where everyone is aware and informed of the outcomes and status. (Vision)

Today we have too many release failures that result in too many rollback failures. If we ignore this problem; resources will need to increase to handle the cascading problems, and we may miss critical customer deadlines which could result in lost revenue, SLA penalties, lost business, and further damage to our quality reputation. (Issue Statement)

Here we see two states for the releases: initial state (low value) tested, and the high value state (in production). There is also an undesirable state (failure) that the new solution will prevent reaching. For me the most important thing is that the problem statement must avoid at all cost to refer to the solution. Even if the people specifying the problem statement have an idea about the solution, they should capture it separately.

This new focus on problem & solution provides a rich conceptual framework to effectively organize the work of a team. After all, we have been innovating, i.e. creating solutions to problems, for thousands of years, so it is no surprise that our vocabulary is quite rich. Here are a few concepts that could be used:

Goal: a goal is not a problem, but you often need to solve problems to reach goals, so it's important to keep them in mind

Fact: a fact often constrains the solution, so they need to be clearly surfaced and accounted for

Assumption: assumptions are very important because they also constrain the solution, but in a more flexible way. Assumptions can be changed, facts generally cannot.

Statement: the problem statement is what physically replaces the user story.

Hurdle: During the decomposition of a problem, hurdles might be identified, they are not a problem per say, but they impact the solution. It could be for instance that a resource is not available in time to meet the deadline.

Snag: A problem can be downgraded to a snag as the solution is obvious to the team and represent a low level of effort. It can also be a small unexpected issue, that need to be quickly resolved.

Dilemma: A problem can be upgraded to a dilemma, when several solutions are possible and it is not clear which one to chose

Setback: The team can suffer a setback when it thought it had found the solution but it didn't, or could not find a solution and need to reassess either the problem or the approach

On the solution side, we can also capture different elements and stages of the solutioning process:

Answer: Findings related to a question raised in the problem statement.

Result: A validation that the solution conforms to a Fact

Resolution: The choice made after reaching a dilemma

Fix: a temporary solution to a problem or a snag to make progress towards the solution to the greater problem

Development: An element of the solution, usually the solution to a subproblem or a snag

Breakthrough: The solution found after reaching a setback

Way out: A solution was not found, nevertheless, the project reached a satisfactory state to meet some or all of the initial goals

...

From a management perspective. The Solution or Delivery Manager can escape the bureaucracy that Agile has created. Ironically, moving stickers around is a zero value activity, with zero impact on the organizational IQ. The solution manager can and should be responsible for the IQ of the project, which rolls up and benefits from the IQ of the organization. It should keep track of the elements that are incorporated in the solution as problems are solved. It should encourage team members to be creative when necessary and to shamelessly adopt existing solutions when it makes sense. It should help resolve dilemmas and push for breakthroughs.

The PMO organization becomes the steward of the Organization's IQ.

As we define problems and solutions in terms of entities, state, transitions and actions, the BOLT methodology provides a unified conceptual framework that spans from Business Strategy to Problem and Solution Domains to Verification (BDD).

To summarize,

1) We have provided a formal model of a problem and a solution, and how they relate to each other

2) This formal model offers the ability to compose problems and solutions at any scale, over the scope of the enterprise

3) Problems and Solutions can be composed from Business Strategy down to Verification

4) We suggest that Agile methodologies replace User Stories by Problem Statements

5) With the renewed focus on "problems", we can also integrate the work of Prof. Knott on Organizational IQ in the whole framework

Last, but not least, decoupling problem definition and solution yields a tremendous benefit in the sense that both can evolve independently during the construction process. 

_______________________________________________________________________________

[1] For instance, you build a vehicle, obviously you want to car to transition to the "in motion" state. Different "actions" will lead to the vehicle to reach that state (a horse pulling, an engine, transmission and wheels, a fan, ...).

[2] BDD Metamodel (Scenario):

[3] Living Social Business Strategy mapped using the same conceptual framework (Source: B = mc2)

I explained last month that the Web Style never took off since Tim Bray's post which predicted the End of SOA with a great dose of exaggeration. Instead, developers quickly pivoted towards a traditional, albeit HTTP-based, API style. I am not sure who could have expected a different outcome... 

Over all these years, the process of defining an API style has been quite chaotic as people swung from resource orientation to service orientation and back and, today, there is no particular "standard" or "sytle" which people can confidently rely on to design their API. Every developer decides artistically (and often inconsistenly) how he wants to use HTTP to marshal a particular API call.

I was doing some research on a few APIs this week-end and took another look at Swagger. I really like what they do. Incidentally, if anybody doubts that the Web Style is dead, I would take a peak at their petshop demo... That being said, that petshop API is terrible. 

When it comes to Resource Orientation, Service Orientation, Event Orientation... I actually like them all: I am a polyadic programmer (a.k.a metaprogrammer) and if anyone doubts it, I would encourage you to look at a framework I built in 2007, WSPER ("whisper"), which stands for Web Service, Process, Event, Resource.  

So, after so many years, should we continue opposing Resource and Service orientation? I would like to show here that indeed we can weave them together and as a result we can surface resource orientation in a much cleaner way while retaining all the benefits of service orientation in the process.

Services have this nice property of being exposed via an endpoint which creates a natural boundary of deployment, access and management. Enpoints truly enable an interface to be totally decoupled from its implementation. I cannot emphasize enough how important that decoupling is to distributed computing.

Service orientation brought another important innovation by moving the focus from objects to views. Services hide, just enough, the model behind the Service Interface: you understand that you are dealing with a customer view, but you never have to understand the extend of a customer class. Actually I would argue that a "true" customer class cannot be built, you always interact with a particular resource in a given context (Sales, Billing, Support...). If you doubt it, please take a look at this presentation from Daniel Jacobson. To be fair, however, as a side effect of this approach, queries or (partial) updates are notoriously painful when you need to define them in a WSDL, no argument there. I get that. But is that a reason to give up altogether on Service Orientation? 

On the other hand the RESTafarians have grossly misunderstood Service Orientation and at the same time, never really understood that REST brought back distributed objects into the programming model. Worse, because of its sole focus on resources, REST forces you to expose every single detail of the model to the resource consumer. It shouldn't be a surprise to anyone that in the end all they can deliver is CRUD. 

One thing that people often forget in SOA is that a service implementation can and should expose multiple service contracts, this is a bit harder to do with APIs because resource paths are not polymorphic by design and they are often hard wired to the implementation. Yes, I know, I could use yet another HTTP header or even concatenate an nth parameter to the Content-type but that is inelegant because you lose the natural boundary offered by a "service" endpoint. The real elegance I see behind Resource orientation (that none of the Web APIs I know implement, including the so-called RESTful ones) is when a path, say /customers/1234, truly points to the same resource regardless of which service I use to access it. 

What I mean here is when you have a customer service and a purchase order service, you should be able to fetch the same customer information (albeit with a different view) using the same path. Yes, when you query an order GET /orders/345 and it returns a path (not a link) to /customers/1234 the PO Service could implement that path and return a view of the customer in the context of a purchase order and if you invoke the same path on the customer service, you would get a more complete view (assuming you have the access rights). 

When you marshal the service name within the path (and the version for that matter), you cannot achieve that elegant behavior without a lot of out of band conventions and string manipulations. That's why I suggest that you define the service/API boundaries using subdomains rather than paths. For instance we would have two subdomains:

http://v1.customers.api.ebpml.org

http://v2.orders.api.ebpml.org

Each subdomain would provide access to its own resource collections, including overlapping collections. Both API calls would return a view of the same customer:

[GET] http://v1.customers.api.ebpml.org/customers/1234

[GET] http://v2.orders.api.ebpml.org/customers/1234

Imagine a "link" that works across services and versions? Many times I have mentioned that REST couples access and identity. Here we have a clear separation between access (subdomain endpoint) and identity (at the path level). 

This API style is capable of supporting the traditional Query/Create/Update/Delete operations quite efficiently, in a way that could look attractive to Web developers. However, I would like to emphasize again that HTTP alone is not a good query language. If you doubt it,please take look at the MongoDB API. Any serious service implementation will have to follow their lead, especially for complex queries and partial updates.

Actions are another point of misalignment between resource and service orientation. Many of the RESTafarians have dismissed the need for actions and advised to use a simple update instead. In the real world this is of course a completely bogus approach (validation, authorization, logging, throttling, versioning, duplication of the update logic on the consumer side...) and Web API designers have been prompt to use action verbs in their paths, wired to specific method calls.

I have also explained many times that every resource has a lifecycle. Some resource (such as a Tweet or Status) have a CUD lifecycle (Create -> Update* -> Delete) which maps well to the lifecycle of a Web page, and hence the HTTP verbs, but again in the real world, lifecycles are complex and each transition maps to an action which you need to express one way or another. A verb is what everyone understands. If you want to use a noun for a verb, be my guest, but semantically you have not changed the meaning of the call, you are still triggering an action, which when successful will transition the resource from one state to the next. 

So there is no reason to be afraid (and do like everyone else): use verbs as needed, either behind a resource when they apply to a resource instance or at the collection level when you initiate a lifecycle.

[POST] http://v2.orders.api.ebpml.org/orders/create

[POST] http://v2.orders.api.ebpml.org/orders/1234/submit

[POST] http://v2.orders.api.ebpml.org/orders/1234/pay

[POST] http://v2.orders.api.ebpml.org/orders/1234/ship

[POST] http://v2.orders.api.ebpml.org/orders/1234/deliver

[POST] http://v2.orders.api.ebpml.org/orders/1234/return

...

Composite calls (which fire several actions that transition to multiple states on different resources) should also be considered.

You will notice that the use of a subdomain as a service boundary greatly simplifies the migration from one major version to the next since API signatures and identities are not polluted by the version number like it is often the case in traditional API designs.

That's it, I believe that approach defines a clean Web API style which brings together the best of Resource Orientation and Service Orientation:

• It achieves a true decoupling between resource access and identity, which in turns allow service consumers to use these identities across multiple services and supports a polymorphic resource access model (at the service level) which encourages the usage of views on the model, rather than a direct access to the model
• It introduces a clean API boundary which can be much more efficiently managed (and authorized)
• It provides a clear and clean way to define actions which map directly to the resource lifecycle
• It simplifies version migrations

I came up with these ideas while building a Swagger stencil for Omnigraffle. So here it is.

While Amazon customers enjoyed something like 26 price drops in that timeframe, I got nothing. Worse, for the kind of servers I use, the price went up 50%, earlier this year: Rackspace no longer offers a micro server (i.e. a slice of a server), for anything I do, I now have to buy a small instance at a 50% premium (after the 30% "drop"). 

So I just got a reserved Amazon's micro instance, a whopping 600 Mb "machine", substantially faster (my guesstimate is about 3X) for about the same price as a now defunct micro Rackspace instance. I can easily run 2 micro Rackspace instances in a single not-so-micro Amazon instance.

Don't get me wrong, I am sad to switch. When I told Robert Scoble that Rackspace's pricing Strategy sucks, he responded and got the President, Lew Moorman and the CTO to reply to me personally. But for me, the cloud is all about slices: temporal slices, or tenancy slices, and clearly Amazon understands that game far better than any other Cloud provider. I am not in the business of sponsoring Rackspace, I expect them to innovate at a reasonable rate and pass that innovation to their customers, the way Amazon does it, or almost any business. Even Apple gets that, the iPhone 4 is 30% cheaper than when it came out, the iPad 2 is 25% cheaper ... and so on. 

The migration was uneventful, a simple file transfer was enough since my workload is stateless. At first glance security seems a lot tighter. 

So long Rackspace.

"SOA isn’t the future, Web style is."

That post opened the door to a big circus and gave pundits the "opportunity" to predict with more accuracy than any oracle the Death of SOA. Anne Thomas Manes explained to us in 2009, from the comfort of her cublicle at Delphi Gartner,  that "SOA [was] a great failed experiment". Another "analyst", David Chappell, added to the circus by asking tong-in-cheek "if SOA was failing", and went as far as calling SOA: "Suckered Once Again". Then there was Jim Webber's "Same Old Atrocity" which is one of the most profound and graph-ical presentation IT had ever seen.

How many SOA initiatives were stopped or slowed down by "The End of SOA"? Mine was. My manager at the time told me that Tim Bray's post was circulating in our IT department and he didn't know where to start to craft an answer to his management. His team had built their own ESB at a time when hardly anyone had heard of XML and several years of hard work and constantly rising transaction volumes (a respectable 10 M requests/day at the time, in early 2007)  were now jeopardized by a single paragraph written by Tim Bray. 

Seven years later, the Programmable Web's directory has close to 9000 registered public APIs (yes APIs, as in Ancient Programming Interface). That number is growing exponentially. Out of 9000 APIs, my best estimate is that less than 1% are following Tim Bray's Web Style. They are all but following an "API" style, short of RPC.

If we just take a sample of the newest APIs on the ProgrammableWeb (as of March 18), we have:

• Ask Ziggy which offers the ability (sic)  to define "actions" (such as Play, NextSong, Previous Song, Shuffle...)
• WhatLanguage explains that you can use, as you wish, a GET (if your request is less than 7500 chars) or a POST to the same URI to detect the language of a given string
• Do.com actually seem to be offering a Web Style API, but it does not do much, it is simply CRUDing 5 resources (tasks, project, users,...)
• SkyBuffer is also following the Web Style, but just like DO.com, this is just some CRUD on a couple of entities
• MaShape which is a "Cloud API hub" is very interesting because they offer a better way for developers to consumer APIs. How do they do that? They invite developers to "Learn how to describe your API on Mashape to autogenerate client libraries and documentation". Yes you heard right, after years of bashing, developers start talking about client library code generation.

I was asked recently to reflect over the last 15 years working on SOA and distill what I learned in the most succinct form possible.

I came up with "the four principles" that will make you succeed at SOA:

1. Service Interface shall be decoupled from Service Implementation
2. All Business Logic shall be normalized
3. Changing a service shall be easy
   • Changes shall be hidden to service consumers until they are ready 
   • Changes shall be easy to consume when the consumer is ready
4. Service Versioning shall be based on Compatibility

 I strongly believe that if you drive your SOA following these four principles, and these four only, you will be very successful at SOA.

Here are the answers to a few more questions that you may ask: 

What is SOA?

SOA is an architecture that positions your organization to manage change in the most efficient way possible.
  

What about Governance?

Don’t “over-govern”, governance should remain minimal and based on common sense with a short term horizon (3-6 months)

Data Governance is far more important since any change to your information model is generally impacting the service interface

What about Cohesion, Autonomy and Service Boundaries?

There is little or no cohesion possible in SOA: Data is relational, you cannot slice a relational model in "autonomous" entities.

When a high degree of cohesion is observed, it is plain common sense, for instance: Google Maps APIs rarely needs to be implemented within your Purchase Order service 

Don’t sweat over your service boundaries, invest in building the best service interface possible (i.e. efficient at managing change)

What is the nature of the “Service Interface”? 

Most people fail at SOA because, they think of a Service as an abstraction, something like a “class” in OO. The Service Interface is a contract that enables changes to be explicit and controled.

How do you define loose coupling?

Loose coupling is achieved when:

• The business logic implemented behind the interface is not involved in managing the context of interaction with a consumer
• There is no duplication of the business logic involved in managing the state of the system(s) of record in the consumers of the interface

What about reuse?

Nobody can expect to build a service today and that service will be ready to be consumed by new consumers 3 years from now. That is ludicrous. If you think of reuse that way you will fail instantly and come up with silly conclusions such as "SOA does not work". 

In SOA (and in the real world too) reuse works the other way around:

• it is not a new consumer who is reusing an old service
• it is almost exclusively a new version of a service (changed to support new consumers) which can be reused by old consumers without breaking them

Do you agree? disagree? Any other question?

 This is a cross-post from my new blog "unRelated". 

The Conway Law seems to be getting a renewed interest lately. In 1968, Mel Conway, then, a manager of peripheral systems research at Univac, devised:

"organizations which design systems ... are constrained to produce designs which are copies of the communication structures of these organizations"

This seminal paper is definitely worth a careful read. Mel's predicates are so visionary that we could easily believe that our ability to create complex systems has not improved in over half a century.

That being said, it might be timely to ask whether the boundaries of the Conway Law are still valid and if its context has changed. 

Insight Before Communication

Leadership tends to focus on the culture and structure of an organization to drive towards desired business outcomes with the expectation is that a better culture and enhanced structure will lead to an effective communication. Actually we are so desperate in our quest to find a better culture that some people go as far as suggesting that "stealing" (ideas) become socially acceptable.

It is interesting to see that in the process nearly everyone rounds off insight, often assuming a perfect ability to gain the correct level of insight into a system design or a question. Mel touches that topic slightly before focusing exclusively on the relation between communication channels and the structural aspects of an organization:

It is a natural temptation of the initial designer to delegate tasks when the apparent complexity of the system approaches his limits of comprehension. This is the turning point in the course of the design. Either he struggles to reduce the system to comprehensibility and wins, or else, he loses control.

I understand that culturally, anyone who questions his or her insight, let alone someone else's insight, will pay a high price for it. We often cover up a deficit of insight as a mere communication disconnect as everyone seeks to appear intellectually sufficient. Mel's predicate is both foundational and consequential because it anchors our perception that insight can't be elaborated and somehow, like knowledge, directly correlates with power. That thought is pervasive in modern organizations where the Sinofskys of the world strive as long as their levels of comprehension and control enables the organization to deliver something. I find it fascinating that 50 years apart, Mel was searching to answer the question "How Do Committees Invent?" and the prevalent culture in corporations like Microsoft or Apple abhors the "Design by Committee" process.

Can we still afford to round off insight? How and what can we really communicate without the proper insight? Don't we spend way more time communicating our insight rather than elaborating it? How many companies fail to benefit from the collective intelligence of their organization?

The Context of Design has Changed Significantly

I don't want to appear condescending to an era of incredible achievements but systems do evolve and new kinds of systems require new levels of insight to elaborate proper designs. Back then, most innovations were product centric. From the mid 80s and well into the late 2000s, innovation became service oriented, and today we are rapidly moving towards an "activity oriented" innovation model.

fig 1. From products, to services to activities

In a product-oriented world, consumers are left to compose individual products to create higher value use cases. In a service-oriented world, businesses identify some of these high value combinations and deliver them "as-a-service" to their customers. The advent of Mobile platforms makes it now possible to design systems that integrate readily with the activities customers are trying to accomplish.

In an activity oriented world, the "edges" of a system take on a disproportionate importance in the design compared to their relative sizes. The focus is no longer on systems and subsystems or even their orchestration into valuable services. The focus is now on understanding every activity customers are trying to accomplish and delivering a set of products and services that will directly integrate with these activities. The design of that integration, that edge, could be, actually, will pretty much always be, far more complex than the design of products and services that support them. You don't design "User Experience" as you design systems from subsystems. Designs need to start from the appreciation of the point of view of the end user, not just from the perceived value of a service or a product.

That change is profound because the design of an edge requires all constituents to cooperate and depart from the unidimensional mindset of product and service centric organizations. Any inefficiency in the scope or variety of edges will have a huge impact on the success or failure of the overall design. In an activity-oriented world, the ability to harness the collective intelligence of your organization directly correlates to your success.

This evolution towards more integrated systems has lead to the emergence of a new homomorphism between the type of insight different groups of people can elaborate and the structure of their organization. Even the very structure of our education system and hence hiring policies are now driven by that homomorphism.

It may not have been true when Mel wrote his paper, but today it is all too common for a group to formulate requirements based their insight to drive the design of other teams. As this new homomorphism developed  the relationship between systems and subsystems, as well as the relationships between products, services and activities made it nearly impossible to devise a structure that would nurture an appropriate level of communication with the goal of achieving a shared understanding.

fig 2. From insight to shared insight

As the levels of shared or aligned insight decrease so does the ability to create compelling and sound designs. Insight gaps develop as organizations tend to focus on building what they can understand as a subset of what their leaders can comprehend. Worst of all, some elements of the design can be built undetected until they fail their organization entirely.

A deficit of insight can be so costly to an organization and its shareholders that it should be accounted for in its balance sheet.

The Tools we Use to Communicate Impact Designs Negatively

The tools we commonly use to communicate would certainly influence Mel's conclusions. Popular knowledge tools (office suite, mind maps, requirements management solution ...) allow us to express ourselves with a greater productivity, but they may also hamper communication much more than we think. Why? The very structure that we use to communicate interferes with our ability to elaborate insight and ultimately designs.

For instance mind maps assume a hierarchical set of relationships, this is unfortunate because most of the relationships in the physical world are not hierarchical. That view seems to be directly inherited from Mel's era where we could still design products in terms of systems and subsystems. Similarly, when we use some "slide-ware" every system looks like a set of layers and when we use "row-ware" such as Excel or any Requirements Management tool to capture and communicate requirements, we cannot effectively represent dependencies, let alone track the elements of the design impacted by these requirements. These artificial knowledge structures are biased and negatively impact our ability to design a system.

Probably, the most important issue introduced by knowledge tools is that none of views that we create are representing any kind of dynamic behavior. How can we create an effective shared understanding between such a wide spectrum of people when everything that we communicate is using the wrong kinds of relationships and lacking even the most elementary dynamic view? Especially when we consider that human languages are poorly equipped to communicate relationships and dynamic behaviors.

A Call for Action

We must consider Insight and Communication separately. Mel's proposition looks attractive:

Research which leads to techniques permitting more efficient communication among designers will play an extremely important role in the technology of system management.

Yet, research that would lead to techniques permitting a more efficient elaboration of insight would have a far greater impact on the system designs.

Organizations should value teams that excel at elaborating insight. This is where education and HR policies should focus.

We have to collectively drive towards closing any insight gap and expand our shared understanding to a level that is compatible with success.

We need new tools and approaches that enhance our ability to elaborate insight.

The path to insight is not as hard as it looks, from federating intuition, to developing perception to growing appreciation and ultimately formulating the vision. Never again, should an individual feel that a group is limiting the design of a system. Never again, should the design of a system be limited by an individual. Never again, should a design reflect only a fraction of the insight of the overall group of designers.

My Favorite Quotes from Mel's Article

there's never enough time to do something right, but there's always enough time to do it over.

there is a homomorphism from the linear graph of a system to the linear graph of its design organization the realization by the initial designers that the system will be large, together with certain pressures in their organization, make irresistible the temptation to assign too many people to a design effort

One fallacy behind [the Accounting theory of management] is the property of linearity which says that two men working for a year or one hundred men working for a week (at the same hourly cost per man) are resources of equal value.

As long as the manager's prestige and power are tied to the size of his budget, he will be motivated to expand his organization.

Probably the greatest single common factor behind many poorly designed systems now in existence has been the availability of a design organization in need of work.

Elementary probability theory tells us that the number of possible communication paths in an organization is approximately half the square of the number of people in the organization. Even in a moderately small organization it becomes necessary to restrict communication in order that people can get some "work" done.

To the extent that organizational protocol restricts communication along lines of command, the communication structure of an organization will resemble its administrative structure. This is one reason why military-style organizations design systems which look like their organization charts.

Research which leads to techniques permitting more efficient communication among designers will play an extremely important role in the technology of system management.

the very act of organizing a design team means that certain design decisions have already been made, explicitly or otherwise. Given any design team organization, there is a class of design alternatives which cannot be effectively pursued by such an organization because the necessary communication paths do not exist. Therefore, there is no such thing as a design group which is both organized and unbiased. Every time a delegation is made and somebody's scope of inquiry is narrowed, the class of design alternatives which can be effectively pursued is also narrowed.

Coordination among task groups, although it appears to lower the productivity of the individual in the small group, provides the only possibility that the separate task groups will be able to consolidate their efforts into a unified system design.

It might conceivably reorganize upon discovery of a new, and obviously superior, design concept; but such an appearance of uncertainty is unflattering, and the very act of voluntarily abandoning a creation is painful and expensive.

A couple of weeks ago, we explored the Future of Social Networks which we see evolving towards:

• friends over followers
• future over past
• activities over posts

In this post we would like to share a couple of the business models that could support these activity centric social networks. But, before we do that let's start by looking at the implications of shifting from a knowledge centric point of view (the Web) to an activity centric point of view (the Platform).

Don't Underestimate SmallData

It does sound a bit trivial to state that we perform a lot of activities with our mobile devices, far more than we ever did with our computers: we purchase our morning coffee, we deposit checks, board a plane, we go places, we take pictures and videos, we learn, we measure our blood pressure, change thermostat settings, select something to watch on TV, open and rent a car ... and that list is not even the tip of the iceberg. Our phone is rapidly becoming a universal remote.

And, even though we are nearly 5 years into the "smartphone" revolution, there is one large area of the Platform architecture that has not been explored: Mobile App Integration. As smartphones enhance every task we do and every action we take, we can expect every task and action to produce invaluable “cues” that could be reused to support other activities from the same user (or from other users) using completely different apps. That's what we call "Small Data", very small data.

We define small data as the data exchanged between two apps that are either under the control of the same end user or an end user and its direct social network (relatives, friends and acquaintances a.k.a RFAs).

The lion share of the Platform's business model is going to be driven by Small Data. Why? Because apps won't need to guess what we want, SmallData is made up of the exact events (location, time and activity)  of our lives and can provide other apps with a wealth of cues that will make them behave exactly as we wish. Most of what there is to know lives on our devices and the ones of our relatives, friends and acquaintances (the RFAs).

For instance, I may need to replace my old washing machine. I can tell my "To Do" app that I need to accomplish that new task (a well defined task). I could even enter right there a price range, features and rankings of the product. You can easily guess the kind of dialog you would have with Siri or Google Now. That information becomes instant context for all the apps I visit next on any of my device.

I can also connect easily the physical environment of my tasks by scanning QR codes or using picture-based database queries.

BigData and Machine Learning algorithms may provide insight over our very next moves, but there is some big barriers that Small Data can help level. BigData cannot guess the tasks we have to do, the problems we have to solve. BigData can only guess what we may want to do. Precision is another big barrier that BigData can rarely address. How many times do you search for something, end up buying it and continue seeings ads for weeks that are trying to sell a product you already own. Imagine how many opportunities to sell complementary products, or simply to ask feedback are lost? SmallData can inform your apps of the tasks you are trying to accomplish and whether you are done buying this washing machine you needed. How valuable is that?

We are at this fascinating moment in time where, from a business point of view, and for all intent and purposes we live in an "unRelated" world, and there is this massive revolution marching (let's call it the SoMoLoCo revolution) which is all about surfacing, realizing and enhancing all the relationships that make up our lives as part of the activities we do. When we say relationships, we mean of course the relationships between people, but also between people and tasks, things or places. SmallData is at the core of that revolution.

The Web Architecture is not optimized for a Task Centric World

In a "related" world, the opportunities to share cues abound but we need a new kind of apps which can exchange seamlessly specific data elements on the same device or beyond, with friends for instance, or with a community. Cues could of course be aggregated across a community and be made available as such.

This exchange of contextual “cues” has been going on, at least, for as long as the Web became commercial, but today, the Web is facing a major disruption that directly challenges its business model: architecturally, the Web always mandates a 3rd party server-side integration to mediate the  exchange of information between two apps (think double-click), unless there is a direct hop from one application to the next (think Facebook's referer).

In the washing machine example, the Web architecture would require either a back-end integration between the To Do app and merchants, or it would require that you always enter a merchant app through the To Do app. Neither are really practical. The problem here is that activities relate to each other in a way that is quite different than simple bits of knowledge, propagating the context of a search to an ad is far easier than propagating the context of a trip to a series of activities. 

That fundamental shift requires a new application integration model as mobile apps are weaved into every task we perform. One could argue, of course, that Amazon could shift from being a "Catalog" to becoming a mundane "To Do" app, but would they have that kind of insight? BigData is in their DNA, they actually don't know much about the tasks customers are trying to accomplish. Before they build that kind of knowledge, there would be a lot of room for alternatives to grow. Other retailers will have to quickly come up with an alternative solution and architecture anyways.

Application integration is a very old idea but today mobile platforms are changing the game. They enable the sharing of cues between trusted users, using trusted apps and mediated by a trusted party. This is somewhat new, trust has never really been part of Web application integration. Quite the opposite actually. You cannot underestimate how much disruption can result from trusted integrations.

So, the key success factor here is to enable the apps on your device to securely and privately share information without requiring a complex temporal integration involving a 3rd party service. The information is produced and consumed on the device or the device of a related end-user. What happens on your device can now stay on your device (sort of).

Just to be clear, and to show how disruptive this architecture is, the primary identifier of your private data becomes your phone. Merchants no longer need to identify you. They can't care less about YOU, they just care to know some information about you. The problem with the Web Architecture was that the only way to do that was by identifying you and tracking your every move. 

This can be achieved with a protocol based on the fact that the device OS (and the Platform behind) is a trusted party for the user, friends, commercial partners… and has the ability to both establish the identity of the applications which share the information, enforce consent of the end-user for the sharing to happen, enrich the integration, report on transaction for commercial applications, …

This is why protocols are likely to be a lot more important than "services" moving forward. The Web has trained us to build dumb clients and centralize anything of value on the server, at a huge cost and never enough trust. We can predict that light-weight protocols, mediated by the mobile OS (and its Platform) will directly challenge the Web architecture, precisely because we can leverage the platform trust model. That evolution is extremely profound.

With all that being said, let's look at a couple of scenarios where we see valuable protocols emerging.

We saw in our last post, with the Starbucks example, that Automated social events, integrated with the tasks consumers are doing are going to be a big part of the future of Social Networks.

There is also an extreme value in decoupling the way cues are captured, stored and shared.  On the Web this is done in a monolithic way, and you have to become a Google AdSense to monetize it. There is a real opportunity in enabling developers to focus on capturing and using cues while the Platform manages and mediates the exchange of these cues in a trusted way. Again this paradigm shift is profound because we are no longer dependent on Google dictating the resulting service (ads) and the information being exchange. A platform based architecture can support many more information sharing scenarios. We actually argue that a trust-based neutral Platform supports a more vibrant and diverse ecosystem than a truly open model.

Platform operators (Google, Apple, Microsoft, RIM...) will enable that architecture because they will continuously seek ways to monetize the trust they established. The paradox is that because the Web is open, any integration between services start by establishing a form of trust and most often that leads to identifying a useful use case (such as AdSense) and mining it, at the expense of many more, but slightly less valuable, use cases.

Share information, Share Revenue

As apps align with the tasks end users are trying to accomplish and as these tasks coordinate several actions, we can expect that applications will see value in exchanging larger number of cues.

The Platform can report on the connection between the usage of a cue (produced by App 1) and a sale performed on App2 in the context of the cue provided by App1. That kind of information sharing mechanism can support a healthy affiliate ecosystem, far more diverse and effective than what AdSense could ever do. So what would the protocol look like? Here is a proposal: 1)  App 2 developer sets up a commercial campaign to promote the sharing of clues with its application 2)  App 1 developer configures his app to provides the clues when end user uses his application.

• Step 2 App 1shares the specified clue.
• Step 3 App 2 queries the clues
• Step 4 End user makes a purchase based on clue
• Step 5 App 2 reports the purchase to the platform (optionally completes the purchase with the platform)
• Step 6 The device communicates the transaction to the platform server (including, campaign identifier provided by App 1 with the clue, identiy of application 1). The platform mediates payouts from developer 2 to developer 1

3) an alternative flow is to provide the cues a 3^rd party service for marketing / research perspective

With such protocol the information never travels to a developer server prior to its consumption, the end user is not "tracked". It is only shared as needed on the end-user device for the benefit of the end-users or with the devices of trusted related end users (RFAs), or completely anonymously, as simple stats.

Social Data integration

We describe here a simple protocol that enables the "social" sharing of cues.

A trusted platform may also share this information with a trusted group of users (for instance friends), either anonymously, or specifically, with the identity of the first user. (steps 5: and 6:). 

The key though is that your friends can now use a completely different set of apps and still benefit from the sharing of cues.

For instance if users open the Fandango application, they might be interested in seeing a percentage of friends who saw or speak about any given movie (anonymously) combining the information from social network posts, iTunes purchases,.... For the Fandago app, this information has a lot of value because it does not need to build that integration, on its back-end, track who is friend with who and who consented to share this information. It would be extremely difficult for every user to manage their social network based on sharing actions such as purchasing a movie ticket or replacing a washing machine.

Social Networks will align with end user interests, but not with every tasks they do. The platform may also aggregate some of these cues and offer aggregated / anonymized information to a group of users. For instance, people checking in at a restaurant or purchasing movie tickets may give a good idea of the wait time.  It may also give local statistics for popular movies, restaurants, events, transportation, …. Cues could also be augmented with the help of a 3rd party service (Step 8) after being produced at step 2 and before they are consumed by step 3. In other words a series of additional clues could be made available to the device OS that neither app 1 or app 2 could infer on their own, just by the user sharing a clue on app1.

With these kinds of protocols, and unlike the Web, the production and consumption of cues has the highest value when the contexts of the apps are connected. A cue-based affiliate model is no longer about delivering “ads” but by making commerce more effective, nearly frictionless, by sharing the right information between the actors that can really make sense of it as users take specific actions. Of course, all this can only be enabled if the end user is given control  over:

• the consent for an application to share a cue with other applications
• the topology of the data integration between applications. For instance a user could be presented with a view of the cues surrounded by apps that would allow to "slide and drop" cues to a particular application.

The very open nature of the Web is driving scale over scope. The Web has successfully nurtured the largest Catalog, the largest Search engine, the largest Auction site, the largest Social Network, but we see this as a drawback because it limits the scope of what people can do. In other words, the scope of what Amazon, Google or Facebook offer is limited by the scale (and hence the revenue) they can achieve. 

The Platform architecture, and what appears to be a closed system, turns that model on its head. The Platform which has already achieved scale can now focus on scope and has the potential to create business models and ecosystems that are far more varied and efficient to solve our every days problems while not compromising our privacy.  The platform could for instance support sharing all our activities' context in the privacy of our devices. We strongly believe that the Web as it stands will not be able to resist that kind of disruption. We expect Business Models will align towards:

• activities over knowledge
• small data over big data
• trusted data sharing over "sniffing around"

Jean-Jacques Dubray

This week-end you can download our book for free on iTunes.

“The Platform” which was pioneered independently by Amazon and Apple over the last decade is creating a highly disruptive environment, introducing a discontinuity at all levels: economic, financial, political and social.

Market structures and Industry boundaries are about to shift in unforeseeable ways, and every space should be considered uncontested. There is simply no amount of historical patterns, cost cutting, dashboards, 4 square frameworks or sustainable evolutions that can prepare an organization to transition into that new world.

Innovating in the Platform era requires a lot more precision and a much higher velocity. It also requires innovating simultaneously at the technology, business and operations levels. 

B = mc2 is a tale of this new world, a multidimensional world, and a unique Business Strategy Methodology, BOLT, that will help you conquer it. BOLT has been designed to enable you to:

- harness the collective intelligence of your organization
- execute at unprecedented speed
- offer a diverse and adaptable set of products and services which can best align with their customers’ needs, just-in-time and just-in-place.

Table of Contents

Introduction

Chapter 1 - The Silent Revolution

Chapter 2 - BOLT
- Introduction
- Systems
- Lifecycle

Chapter 3 - Formulating a Strategy with BOLT
- Introduction
- The CEO’s Dilemma
- Platform-Driven Product Strategy
- Market and Sell to the Lifecycle
- Business Integration
- Conclusion

Chapter 4 - Executing a BOLT Strategy
- Introduction
- People
- Organization
- Technology & Information
- Conclusion

Conclusion
Acknowledgments