Chaotic IT

We've Moved...

2009-09-08T21:28:00.000-07:00

The sun has set on Chaotic IT. I am now posting on The SOA Coach blog. Same content, stickier name. Hope you can drop by.

Message to Corporate Dinosaurs: The Meteor Has Landed

2008-08-01T20:13:00.000-07:00

65 million years ago something unexpected happened that altered the course of Earth's history forever. Dinosaurs had reached the apex of evolutionary fitness and had ruled over land-based animals for millions of years. They had become indomitable masters of their surroundings. Then, suddenly, in a geological instant, they vanished.

Geologists and paleontologists call this the "K-T event." Whatever happened 65 million years ago was so devastating that it caused an abrupt change in the fossil record, marking the transition between two major geological eras, the Cretatious (K) and Tertiary (T). The K-T event wiped out half of all land-dwelling creatures on the planet, eradicating the strongest, fittest, and most powerful species but sparing many smaller, weaker ones. Among the weaker ones were mammals, which of course eventually gained dominance. What is most curious about the K-T event, or any mass extinction, is why inferior species survive while superior ones perish.

The answer is agility, and the fit often sacrifice it in favor of sheer mass.

The dinosaurs as a species did not turn to dust in an instant. Rather, they gradually died off over a period of 10,000 years or so, most likely of starvation in the wake of severe global climate change. All creatures were subject to these conditions of extreme scarcity, but the dinosaurs were at a distinct disadvantage because their size required them to find comparatively more resources to survive. From an evolutionary standpoint they got careless and allowed their size to compromise their ability to adapt in the face of change. Meanwhile, millions of smaller species thrived because of their much more economical consumption of energy. I believe we are witnessing a similar extinction event in today's business ecosystem.

The untouchable corporate monoliths of the industrial age may be headed toward a similar fate as the dinosaurs, threatened by the global impact of the Internet Era. For these 8000-pound T-Rexes, what used to be uncontested dominance over resource-abundant business territories is withering into a struggle to fend off swarms of new and nimble competitive breeds encroaching from all directions. The digital age has declared open season on traditional business models and has little appreciation for historical aristocracies. It's anyone's game, and everyone is stepping up to compete.

Michael Hugos of CIO magazine blogged about this phenomenon, citing how Kodak, GM, and Ford have sustained deep wounds in the face of new competitive threats. He says of today's corporate heavyweights,

"They are rigid hierarchies through and through; it’s in their genes; it seems they are not and cannot be agile."

If this is indeed the case, then these companies' existence is endangered. Today's business environment increasingly favors organizations that can self-organize quickly and dodge the effects of constant change. Heavyweights cannot maneuver around change, so they become big targets that sustain constant damage. Lacking the ability to adapt, time will eventually erode the pedestals out from underneath them.

Some heavyweights are attempting to inherit agility through acquisition, such as News Corp. via its jaw-dropping purchase of MySpace. Yahoo! and Google (yes, even they are threatened) have strategically acquired web-based social networking prodigies Flickr and YouTube, respectively, to remain a step ahead in the game of evolution.

No industry is immune from this arms race of innovation, though. It is corporate Darwinism on an epic scale. Newspapers, movie companies, record labels, retail outlets, auto makers, schools, restaurants, every business is affected by today's rapid pace of technological change. And many large, market-dominating predators are most at risk because of the sheer magnitude of effort required for them to suddenly do things differently.

I believe we are on the cusp of a massive corporate extinction event similar to the K-T event. The blast has struck and it will take years for the dust to settle. Meanwhile, the chaos has disrupted the economic balance of power, offering new advantages to the small and nimble. Like the dinosaurs, large companies may not perish overnight. But for those not able to adapt, the death knell has already begun to toll.

Long Tail SOA and the Mythology of Reuse

2008-05-04T20:18:00.000-07:00

I have been preoccupied for the last several months with traveling and becoming a new dad, but I managed to cobble together an article for the latest issue of SOAWorld magazine. It presents and contrasts two rather opposed SOA strategies--Mainstream SOA and Long Tail SOA--and argues that organizations embracing SOA tend to focus too much energy on building Services for reuse. Highly reusable Services perform business functions that are in high demand, hence the "mainstream" label, but these Services are scarce compared to the accumulated demand for one-off Services. In general, only 20% of a company's Services are highly reusable. Targeting only these neglects the other 80% of the company's Service market. To truly become agile, IT organizations should build Service portfolios at the Long Tail end of the spectrum, because Services do not need to be reusable to be valuable. . .they just need to be available.

http://soa.sys-con.com/read/557348_1.htm

How to Ruin a SOA Program and Bankrupt IT

2007-07-01T15:51:00.000-07:00

I spent the first half of last week in New York city at the SOAWorld 2007 conference, where I had the privilege of presenting my thoughts on the ROI of SOA. It was a fun session with a very inquisitive audience. I have received a lot of valuable feedback on the presentation and the supplemental article in SOAWorld magazine and would like to address one topic that has come up frequently, and that is whether simply choosing SOA over point-to-point (P2P) integration is enough to achieve agility.

I explained in my article that a fundamental characteristic of SOA is its linear cost curve and contrasted it with P2P's non-linear cost curve. The "bottom line" is that SOA inherently keeps costs linear, predictable, and scalable over the long haul while the cost of P2P accelerates uncontrollably. The net effect is that service-oriented networks are enterprise assets that appreciate in value as they grow, while P2P networks are depreciating liabilities. SOA, by its nature, produces positive ROI. P2P, by its nature, produces negative ROI. Therefore, if you embrace SOA and forsake P2P you should be set, right? Wrong.

To be sure, taking a stand against traditional P2P integration is a big leap in the right direction. However, it is one thing to say you are service-oriented and quite another to be service-oriented. And if you're talking the talk but not walking the walk you can end up in worse shape than if you had opted against SOA in the first place.

SOA often requires a paradigm shift -- a shift in attitudes and thinking away from project-based, short-term objectives toward enterprise-based, long-term objectives. When the people doing the talking are not the ones doing the walking, the mental shift may not occur everywhere it needs to. This can produce a situation where an organization has obtained sponsorship and funding for an SOA program but lacks the discipline to adhere to its core principles during implementation. When SOA has been promised but the implementers are only comfortable with P2P, the result is often point-based SOA.

From a financial standpoint, point-based SOA combines the short-term cost hikes of SOA with the long-term, accelerating cost accumulation of P2P. What is left is an overly complex network of tightly-coupled services linked together through physical connections and proprietary interfaces. (Imagine a network of web services in which each service can be used by only one consumer.)

Relating this phenomenon back to the Bottom Line analysis, the resulting cost curve (P-SOA) resembles the P2P cost curve. The only difference is that it has been shifted upward by a coefficient that represents the extra infrastructure costs of SOA. In the end, point-based SOA costs more than traditional P2P integration and yields no gain in IT flexibility or agility.

IT organizations that are adept at selling SOA at the podium need to also have the tactical ability to execute on SOA missions. This requires shared vision, passion, and deep commitment to the fundamental principles of SOA at all levels of the IT ranks. Companies that have difficulty producing this cultural unity are at a distinct disadvantage in ever realizing the benefits of SOA. In fact, they may be better off just ignoring the hype and walking away.

In Defense of Bottom Line SOA

2007-05-15T17:36:00.000-07:00

Just when I thought I had sold the SOA community on my perspectives on the ROI of SOA, Alastair Bathgate, a fellow tech blogger whom I respect, threw down the gauntlet and took some jabs at my recent whitepaper. Two thoughts filled my head as I read his retort:

He's right
I'm glad he spoke up

His first point:

"Although, as Marc points out, the cost of connecting all the components in a P2P architecture is undoubtedly high, I wonder if that would ever happen in reality..."

In my paper, I based my ROI analysis on the assumption that total infrastructure connectedness is required to achieve maximum agility. However, it would be foolish to think that every last component in an IT environment would need to be connected to every other in order to produce appreciable return on integration investments. In fact, it may be altogether impossible. However, I do contend that business-critical segments of the infrastructure can be configured into fully connected integration sub-networks.

Bathgate continues:

"[Marc's] target of zero latency...is again appealing, but unlikely to be achieved...even with the most sophisticated SOA."

Again, he's right on the money. No company can ever hope to operate in absolute real-time, even with a perfectly managed service-oriented infrastructure. There are countless forces at play constantly that contribute friction to even the most streamlined business processes. (It is these forces that produce the chaos that is the theme of this blog.) In order to home in on the essence of SOA and P2P, though, I had to eliminate these variables from my equations, otherwise my analysis would have been so muddied up with extraneous data that the fundamental properties of P2P and SOA wouldn't have shined through.

Then he adds:

"...the enterprise is always dreaming up new business processes...which means that maintaining a Service Oriented Architecture can be just as painful as building P2P solutions."

"...the orchestration layer of any SOA needs constant reconfiguring to meet new business requirements."

SOA certainly can be just as painful, or more, than the P2P approach (I assume that by "painful" Mr. Bathgate means costly), but this is a function of the quality of a particular SOA implementation and is not an immutable property of SOA itself. Some SOA initiatives will succeed and some will fail. That's just the way it is, but we shouldn't fault or avoid SOA just because success is not guaranteed.

Also, I agree with Bathgate on the point that SOA is not immune to change. In fact, my ROI model is based on the assumption that service-based networks are subject to at least the same rate of change as P2P networks. The bottom line, though, is that SOA isolates an instance of change to a single location rather than allowing it to rip through the network like a shock wave.

Finaly, Bathgate closes with:

"I think that Marc has written a highly interesting and thought provoking paper that considers the “pure” arguments. He points to a fork in the road where a decision must be made to take the P2P or SOA paths. I just think that in the real world there is some dirty middle ground that we have to accept exists..."

Again, I could not agree more. My ROI model is a starting point for understanding the raw characteristics of SOA as compared to P2P. It is not meant to be construed as a methodology or formula for success. It places SOA and P2P into a vacuum, free from the influences of external, real-world forces, systematically dissects them and analyzes their DNA under a microscope. Indeed, it is a highly purified view of two extreme approaches to enterprise integration. Perhaps overly scientific for many practical purposes, but necessary to reveal their most fundamental characteristics.

Mr. Bathgate has reminded us that enterprise system integration is not a black-or-white affair. It's not all or nothing. It's not P2P or SOA. In reality, every IT organization must strike a balance between the extremes -- a blended strategy that applies the right philosophy to the right context at the right time. Use my ROI model to understand where the boundaries lie and navigate your own path to agility using sound judgment and an understanding of your business that could not possibly come from a paper you downloaded from the Internet.

Gartner Bullish on SOA (Tell Us Something We Don't know!)

2007-04-25T21:14:00.000-07:00

Once again, Joe McKendrick out at ZDNet published a very quotable blog article. He cites a few notable sentiments from Gartner research director, L. Frank Kenney, about the outlook of SOA and throws in a few of his own.

Kenney suggests firms "aggressively invest in SOA as it will rapidly become the architectural foundation for virtually every new business-critical application." McKendrick adds that, according to Gartner, 80% of mission-critical operational applications and business processes will be SOA-driven by 2010. (I assume these are IT solutions as opposed to "over-the-counter" ones.)

This message shouldn't shock anyone who has been following Gartner's SOA forecasts over the last few years. But what is unique about McKendrick's article is its cautionary tone against using Gartner's bullish outlook as an excuse to go on an SOA spending spree. He states that "the downside [of SOA initiatives] is that too much is being expected of SOA in too short of a time" and adds a few more quotes from Kenney supporting the notion that it can take a long time, years even, for SOA to show appreciable ROI.

The lesson here is that companies should be very careful not to run up their SOA tabs simply because SOA is becoming big business. SOA is an investment in the future of the company. Like any good investment, SOA will operate in the red for a while before it matures and begins to show real returns. (I've never heard of a passenger jet that paid for itself on its maiden voyage or a brand new office high-rise that broke even before all its floors were built.)

Companies should invest in SOA, but not foolishly. A dollar spent on SOA today is a dollar that won't be seen again until sometime after our next president is sworn in. It's a realization that every SOA stakeholder should be comfortable with. So, by all means, go shopping. But take a list! Know exactly what you're buying and how long it's going to take to pay it off.

(See my recent whitepaper for more of my thoughts on how much SOA really costs.)

Free Whitepaper on the ROI of SOA

2007-04-23T21:38:00.000-07:00

One of the most difficult aspects of Service Oriented Architecture (SOA) is demonstrating tangible return on investment (ROI). Many folks concede that SOA is more of a long term value proposition than traditional architectural approaches, but that doesn't do much to lower the blood pressure of the business sponsors footing the bill. SOA programs need strong business backing to be successful, so SOA advocates need to be able to convince the business, and keep them convinced, that it is the right way to go.

I recently wrote a whitepaper that compares the fundamental economics of SOA versus classic point-to-point enterprise architecture. It puts a few simple but powerful concepts and mathematical formulas behind the oft discussed but seldom quantified differences between the two disciplines. My goal was to offer the IT community an objective analysis of the economics of SOA and a simple model for understanding and tracking its ROI over time. These concepts have helped me through many strategy meetings and I hope they can do the same for others. Here is a link to the free download (no registration required):

Bottom Line SOA: The Economics of Agility

All I ask in return for the free download is a bit of constructive feedback, either as a comment on this blog or directly to me at marc@marcrix.com. Also, this paper will be appearing in the June issue of SOAWorld magazine and I will be presenting it in person at the SOAWorld 2007 conference in New York City in late June. I look forward to your comments.

Chaotic Innovation: Give Mavericks the Keys to the Castle

2007-02-12T08:00:00.000-08:00

This post is inspired by a recent article by Joe McKendrick, one of the more fair and balanced SOA bloggers in the 'sphere, entitled, "Is 'rogue' IT always a bad thing?" He points out that, on occasion, employees outside the ordained chain of IT command are guilty of developing ad hoc technology solutions that do not conform to prevailing corporate standards. In the two examples he provides, the rogue solutions provided clear benefits to the business. However, recognizing that such vigilantism by maverick techies runs the risk of undermining important governance controls, he asks his readers whether IT departments should crack down on this behavior. This question cuts to the heart of my theme of Chaotic IT, so I'll proudly offer my take, arguing that IT should not only allow it but encourage and reward it.

First, we must understand that businesses are complex systems; each is composed of myriad independent "parts" (employees, information systems, teams, customers, partners, and so on) that work together to meet business goals. What makes them complex is their endless variety of parts (e.g., no two employees are the same) and the sophisticated web of relationships they make up. This inherent complexity makes the behavior of the business largely unpredictable, and generally uncontrollable by those relatively few parts in leadership positions (just as ant colonies or bee swarms are not controlled by their respective queens). When we are at peace with this, we are at peace with the fact that no one individual, or group of individuals, runs our businesses--we all do, collectively.

One important business theme of the modern era is agility. We must innovate and adapt faster than ever before because, if we don't, our competitors--or a twenty-year-old with a wild hair and a Ruby on Rails book--will. So how do we make our businesses more fluid, adaptive, responsive? It would be great if the answer came in a box. Then we could simply buy it, plug it in, go to the beach, and giggle into our Treos as we watch our stock blow the roof off of Wall Street. But, alas, there is no shrink-wrapped silver bullet. (Any CIO that has invested in SOA technology should know this first hand by now.) The good news, though, is that many of the resources and infrastructure companies need to become agile are already baked into their businesses. They just need to be discovered and exploited.

Some notable thought leaders lately have put forth the concept of initiating "change from within" or "inside-out" transformation to achieve new levels of competitive advantage. While the terminology differs slightly among them, the idea is consistent: that the solutions to a company's strategic innovation challenges already exist within the organization, most likely within the minds of employees down in the tranches, not the leaders at the helm. Whereas the leaders may have a mental picture of the solution and an inclination to spark a project to bring it to fruition, employees at the front lines may already possess finished, tested products that are ready to put before real customers. It stands to reason that if this is indeed the case, companies are better off seeking out this innate knowledge and cultivating it rather than attempting to invent it from scratch. Tim O'Reilly calls these key individuals "alpha geeks" that exist at the "edges" of their companies. Malcom Gladwell, in his book The Tipping Point, calls them "Mavens." Richard Tanner Pascale and Jerry Sternin, in a May 2005 Harvard Business Review article, call them "secret change agents" and "positive deviants." (It is this last term that, to me, resonates most closely with McKendrick's notion of "rogue" engineers.) These authors regard individuals outside mainstream IT as critical to the success of corporate change initiatives, even though the knee-jerk reaction to their efforts often is to apply negative labels to them, such as "deviant," "rogue," "vigilante," "subversive," or "illegal." These individuals, while they seldom occupy positions of high authority, have a deep understanding of both technology and the business drivers that fuel innovation. What's more, they exist at all levels of the organization and can significantly outnumber the mainstream leaders. It is as if every company has a secret underground reservoir of extreme innovation talent that, when tapped, can bring about significant positive changes in agility levels. By leveraging the efforts of these so called mavericks, a company, at the very least, is able to cast a wider net over its sea of innovation possibilities.

Finally, I would like to add my own point of view to this topic--the chaotic take. As I mentioned before, businesses are complex systems. As such, they behave much more like living organisms than machines and are constantly influenced by forces beyond anyone's direct control. (See my previous post entitled Snowball Oriented Architecture in which I introduced the concept of corporate physics. I'll also expand on this in more detail in future posts.) Since businesses exhibit the fundamental characteristics of living things, I look to the natural world for clues about how businesses--and their IT departments--may operate more competitively. One observation I have made is that living things--all of them-- are inherently agile. If we learn what makes them agile perhaps we can apply the same principles to our businesses.

From cells to bacteria to people to herds to flocks to swarms to entire ecosystems, all living systems, like businesses, must undergo constant transformation in response to changes in their environments for the sake of maximizing their odds of survival. It's a game of evolution, no less real for corporations than it is for Galapagos finches or the influenza virus. We all play by the same rules.

So what makes living things naturally agile? It is simply the ability of their constituent parts to rapidly self-organize in the face of some external threat. This is a two step process. First, the threat must be detected quickly. Second, a positive transformation must take place quickly to counteract the threat. The first step is made possible by the awareness of each part to its immediate environment, and the second by its ability to react without first obtaining "permission." For example, when a person touches a hot burner on the stove, sensory cells in the hand register the threat and react immediately by sending an electrical impulse to the brain. Also, in order for evolution to work, individual strands of DNA must be allowed to mutate on their own, without direction from the brain or other external source. These small, local deviations from the norm are then detected by nearby elements and propagated if beneficial or rejected if not. This cycle of "selection" continues until all components have been transformed or the variation has been eradicated, depending on its overall usefulness. The important things to remember are that 1) adaptive changes are nearly always initiated by components at the edges of complex systems, without the knowledge of a central source of authority and 2) that positive change sweeps through these complex systems from the bottom up.

So, "rogue" initiatives are healthy, natural processes that foster a spirit of constant experiementation and reinvention within our businesses, which is crucial for building agility. Sometimes the results of this experimentation will be beneficial, sometimes not, which is why governance models should incorporate feedback mechanisms to discover and evaluate the "fitness" of innovations that flow from the bottom up. In my opinion, this practice is much more contructive than cracking down on rigid policies that keep the flow of creativity traveling down a one way street from the top down. So empower the mavericks, turn them into sensory receptors and change agents for the business, and let them know that the leaders are listening.

I didn't originally intend for this post to be so lengthy, but I believe the decision to allow or disallow "rogue" innovations will have a critical impact on most companies' quest for agility. It may seem counterintuitive to grant employees the autonomy to act according to their own judgment based on cues perceived only by them. In fact, some would probably say that it invites chaos into the business. My point exactly.

SOA: Snowball Oriented Architecture

2007-02-05T08:02:00.000-08:00

I don't consider myself a cartoon freak, but I have always had a warm place in my heart for them. I was practically raised on Loony Tunes and today I can't get enough of a disturbingly addictive little show called Happy Tree Friends. In one recent podcast episode, From Hero to Eternity, was a scene where a cute little chipmunk, disoriented after almost being blown up by a flying squirrel, stumbled off the edge of a cliff. She landed on a steep, snow-covered hill and tumbled down it, gathering snow as she went. Faster and faster she spun until she was encased in a snowball. The snowball soon became gigantic, sucking every living creature in its path into its swelling core, and threatened to pummel a small, hapless village full of cutesy mountain animals. I'm no buzzkill so I'll stop there with the details.

Now, I've seen this "snowball barreling down a hill" bit a hundred times in cartoons. (Who hasn't?) But I grew up in Colorado, so I know a thing or two about snowball physics, and it just doesn't work this way in real life. Anyone who has tried this knows that if you push a little snowball down a hill it does not speed down a straight path and become a perfectly spherical, gigantic version of itself. What really happens is that the little snowball gathers snow in odd places as it's rolled, quickly becomes lopsided and pulls in one direction. When it picks up enough mass to roll on its own, it looks more like a lumpy log than a smooth snowball. At this point, gravity and inertia take over and it's anyone's guess where the thing will ultimately end up and what it will look like when it gets there. So all those cartoons ignore the subtle, cumulative effects that various environmental conditions have on the outcome of the event - things like the stickiness of the snow; the steepness and shape of the slope; the shape, weight, and speed of the snowball; friction, wind, and even the outside temperature. In the cartoon world things are simple and predictable. In the real word they are very complex, even for things as seemingly simple as a snowball rolling down a hill.

I believe many people involved in Service Oriented Architecture are doing the equivalent of pushing a cartoon snowball down a frictionless slope, trusting that if they form a small initiative and give it a nudge in the right direction it will blossom into large-scale success under its own momentum and meet business objectives dead-on at the end of the journey. However, in reality, all change initiatives, SOA in particular, are subject to the constant effects of "corporate physics" - politics, culture, skill sets, methodologies, technology, sponsorship, governance, communication, vendor relationships, budgets, strategy, and so on - just as real snowballs fall under the constant influence of physical laws of gravity, motion, friction, and chemistry. Additionally, unexpected subtleties of any of these effects can lead to dramatic shifts in direction. For instance, a rock buried beneath the snow can throw a rolling snowball wildly off course. Likewise, an unexpected budget cut, missed deadline, or overlooked requirement can trigger a chain of events that can jeopardize an SOA program. The only way to ensure a rolling snowball stays its course is to follow it down, nudging it back on track and reshaping it when it begins to stray. Similarly, the only way to ensure SOA stays its course is to follow it every step of the way, be tuned in to the laws of corporate physics, and reshape the initiative as chaotic events begin to derail it.

SOA holds a lot of promise for companies striving to build agility; it is designed to harness chaos and use that energy to fuel innovation and rapid change. However, before the promise can become a reality companies must build suitable, change-ready SOA infrastructures, which takes patience and careful planning. The keys to successful SOA implementation will be thorough understanding of its fundamental principles, attention to how it influences, and is influenced by, corporate physics at each stage of its evolution, and a deep commitment to following through. Otherwise, the SOA program runs the risk of getting stuck or, worse, barreling out of control through the organization causing mass hysteria and flattening innocent co-workers.

Let the Chaos Begin

2007-02-04T20:00:00.000-08:00

I believe most of us by now are in violent agreement that the shape of business is changing for the 21st century. This is an age of rampant unpredictability where stars are born (and fizzle out) overnight. No one knows when the next startup with a 20-year-old CEO will invent an entire industry or when the next stalwart 20th century empire will crumble. But now we all know that it is possible (if not probable), and that it can happen anytime, anywhere. Hence, we are all faced with a choice: to embrace and attempt to leverage the chaos or to ignore it and hope it blows over.

This blog is dedicated to those of us who have chosen to face the beast head-on, those of us who believe it is here for a reason and is here to stay, those of us who are compelled to study it with the goal of understanding it, taming it and, ultimately, harnessing its power to use to our advantage. We understand that its power is fierce and nondiscriminatory, and that this presents despair for the meek but opportunity for the bold. We understand that we brought this chaos upon ourselves through our own innovation and that its path will be influenced by our continued innovation. And as innovation requires change--breaks from the norm, divergent thinking, non-conforming attitudes, creativity--we are the change agents, and with the power of change at our fingertips we will move mountains.

In this blog I will explore the immutable forces of chaos acting upon every company and attempt to offer insights into how this chaos can be leveraged to defy the threats it imposes and capitalize on the opportunities it uncovers. Those individuals that understand these forces will be poised to become heroes, and companies poised to rocket to the top of their markets. Those that do not will watch in bewilderment as their careers and businesses (slowly or quickly, but assuredly) burn out and fall to the ground.

The playing field has been leveled and we all have been given uniforms but we must each decide for ourselves whether to play the game. I hope you'll suit up and stay with me as I explore this fascinating new world of unprecedented uncertainty...and opportunity.

Let the chaos begin...