There is a lot of misconception rather than confusion on CEP and how it relates to other technologies. I think all the three technologies - Business Rules, Business Processes and Complex Event Processing were developed independently and so there is a lot of overlap in functionality and when they are now being used in combination, each will be used for things which they do best.

Rather than believing that CEP only detect, filter, analyze and correlate diverse business events, it is better to say they do this best and should be used with BPM for workflow management and BRE for decision management.

I did a post on A representational Analysis, here http://blog.bizense.com/2008/12/27/a-representational-analysis/ , which briefs how the representational power is being increased by using combination of business rules and business processes.

Actually, there was a time that I wrote a banking rulebase for processing that used four different analytic methods (champion -challenger being one of the) and then four different stressing methods. It was quite complex in that once the stressing method had been selected then the analytic methods were run again to be sure that that was still the best method. But, it worked quite well.

I think that analytics is something that rule guys forget - that there is much more to writing business solutions than just solving a bunch of "IF - THEN" rules. There are complex strategies depending on data, where collected, TV markets, competition, etc., etc. All of that CAN go in the rules but true forecasting methods are usually outside the expertise of rulebased guys who have failed to understand the simplicity (in comparison) Rete Algorithm.

SDG
jco

Fuzzy definitions only help technology vendors – who can hope to benefit from the confusion to promote their solution versus the ones from the competition – and the analysts and industry gods and oracles – who can hope to benefit from this very confusion in order to promote their clarification services.

The user community ends with the burden of having to figure out what hides behind the definitions as used by each solution, how to best leverage each one of these solutions, how to combine them, which one to use for which purpose, etc.

Having all the technology pieces provided by a single vendor is no panacea to this problem. These single vendors tend to be large, tend to provide offerings consisting of multiple technologies from varying sources (read acquisitions) that also overlap, and introduce confusion – and I won’t even go into what happens trying to get architectural and technical support.

I think Carole-Ann’s blog is right on target with respect to the definitional boundaries for the clear definition of responsibilities assigned to the various technologies and her analogy is clear.

To elaborate a little, my opinion is that a large part of the debate comes from two key and related issues:
- The domain CEP wants to cover is too wide and overlaps with multiple well established domains
- The CEP terminology is awfully confusing

Carole-Ann covered the first, let me cover the second.

a. “Complex” is absolutely not a good term to use in a definition such as this one.

The definition of “complex event” from Luckham is “It is an event that could only happen if lots of other events happened”.
The problem is that in “Complex Event Processing”, it is often the case that “Complex” is applied to “Processing” more than to “Event”. And this confusion then begs questions such as: “Complex as opposed to what? What is the threshold between simple and complex? Is it the same for all problems and problem areas? Does “complex” describe what the problem to solve is, what we – vendors – need to deal with in the resolution of the problem, or what the user community end up dealing with?”

I do not recall – and I count on the blogosphere to prove me wrong – any such definition that establishes itself using a qualitative adjective such as “complex” at its core.

BRMS, BPM do not qualify their focus in such relative scale: they qualify it on an explicit business objective and a role (or group of roles) they cater to: management of business rules and processes, respectively, and the business management community.

b. “Event processing” is not that good a term either. It’s incredibly wide, in particular because it conveys neither intent nor scope. That creates confusion.
This leads, for example, to the assessment I read somewhere that a Loan Origination application is a CEP application, because it consists of the “complex processing” of the loan application “event”. Of course.. Any request for anything can be converted to the event that the request for that something occurred. And anything you do in response to the event occurrence can be qualified as “complex” (it certainly is for somebody) “processing” (since it probably consists of something else than a NOP).

I do agree with Tim’s comment earlier that it’s not bad to have experimentation and that competition and evolution will bring the best definitions forward. And I would add that is precisely what we are trying to do through this effort...

I do not think the BRMS or BPM industries are protecting their respective turfs in this discussions. We all know products bleed one way or the other. But long gone are the days when BPM vendors would claim they would do BRMS and the contrary – and the industry is better for it.

Just to add to this subject, I posted a little note on IBM's recent BEM announcement in http://architectguy.blogspot.com/2008/12/ibms-bem-announcement.html. More oil on the fire :).

http://it.toolbox.com/blogs/the-soa-blog/eda-cep-bpm-bpms-and-soa-28565

I agree that there is a confusion, and that resolving it is quite simple. See: http://epthinking.blogspot.com/2008/11/on-basic-classification-of-terms.html
for further clarification of this issue.

cheers,

Opher

I fear that I must take umbrage with Mr. Raden. (I have ALWAYS wanted to use that statement and this just seemed like a good time to do it.)

Proper definitions are what gives us a firm foundation upon which to build better and more clearly defined tools and paths from one tool to another, a way to set up boundaries between tools so that one tool cannot claim to be that which it is not, and a way to properly define expected behavior.

An automobile is NOT an SUV is NOT a motorcycle is NOT a bicycle. An SUV is a subset of an automobile while a motorcycle is a subset of a vehicle. These do not, normally, need definition since they are what someone in AI would call a "common sense" understanding of the definition. However, without clear definitions we drift into confusion, chaos and the eventual madness of marketing machinations.

Sorry Neil. You blew it. Maybe your systems should be just a bit smarter. :-)

In contradiction to both you and Carole-Ann, I have never felt that we could equate Complex Event Processing to work flow, business processing (managed or otherwise) nor to biological events. Neural Nets are probably the closest thing that we have to simulating the nerve connections to the brain, a true CEP system. Having worked with the analog versions of such things as well as the binary simulations, I do know that we have nothing today that comes within 10^-15 of the power of the human brain.

Complex Events Processing (CEP) has been defined on Wikipedia as "... primarily an event processing concept that deals with the task of processing multiple events with the goal of identifying the meaningful events within the event cloud. CEP employs techniques such as detection of complex patterns of many events, event correlation and abstraction, event hierarchies, and relationships between events such as causality, membership, and timing, and event-driven processes."

Now, whether we agree or disagree with this definition, we should at LEAST try to come up with an industry agreed-upon definition so that the casual observer would NOT be disillusioned nor led astray by some unscrupulous sales person. Also, there are are many, many more definitions that need standardization in our industries. BRMS was only clarified in 2003 - 2005 because ILOG, Fair Isaac and InfoWorld insisted upon a clear definition.

Now we have Drools coming out with a really strange definition of their version 5 BRMS product which is nothing more than a web interface, repository and GUI all rolled into a single application. Somebody needs to slap their collective wrists for not having taken the time to look around at other definitions before really screwing up the market place with an absolute misnomer.

SDG
jco

I actually agree that it is better to solve real world problems than to argue over descriptive semantics. It is really hard to describe a vast body of knowledge with short catchy phrase like "Complex Event Processing" or "Decision Managment" and, by the very nature of humans, there will be a group, or tribe, which will defend their turf with all the energy they can muster. This is true, not only of marketeers, but of humans in general.

There is certainlhy merit in Carole-Ann's view, but I am not sure I fully agree either; but to be fair, I do not strongly disagree either. I can definately see Carole-Ann's perspective; however, I can see the other view that we are creating artifical boundaries. There is actually nothing wrong with overlap and competition.

I agree that confusion is not good, but it is generally not "confusion in terms" that tends to impede progress as much as the lack of user acceptance. After all, the WWW and HTTP simply work and users like it, because good working solutions always prevail. Endless debates tend to happen when the users do not adopt the technology so the sellers must reposition and try to convince users to adopt.

This is exactly what is happending in the CEP space, IMHO. Folks are not adopting CEP, as envisioned, to "detect business opportunities and threats in real-time" because adding another rules engine type of technology to the solution set(s) does not provide users the additional capabilities they need to solve real-world complex problems. Users already have rules engines, and rules engines are quite mature, from a technology perspective.

The arguments and debates are not coming form the buyers as much as they are coming from the sellers. The buyers have already voted, and they are not buying the hype.

Yours sincerely, Tim

If we all let it slip and use our own language without common agreement on what we mean, we will mislead end users and get them to struggle the same way they have struggled when we did not help them enough differentiate BPM from BRMS.

Forrester is working on such a paper too. Knowing that they typically provide those analysis as a result of the questions / demands they get from their customers (which are the real end users), it seems that this exercise has more value than some academic argument on my term versus your term.

Do not underestimate the power of confusion. It lead to decades of dark ages for artificial intelligence. End users hate to be mislead and disappointed. I would not blame them.

Carole-Ann Matignon
VP, Product Management at Fair Isaac

These kinds of definitional arguments are sort of tedious. Why can't there be more than one definition of CEP? Who cares? Marketing people, that's all. The question is, do they work and can they solve a problem that needs to be solved? Why not spend more time articulating that so we can put this stuff to work.

-Neil Raden
twitter: nraden

I just came across the Oracle announcement of its entry in the CEP world: http://www.ebizq.net/news/10671.html?rss

I quote: "The solution is designed to enable organizations to detect, filter, analyze and correlate diverse business events".

Carole-Ann Matignon
VP, Product Management at Fair Isaac

I expected some reaction. This is ECA (Event Condition Action) at work ;-)

I am pleased that we actually came to the same human body analogy independently. This validates that we are not so far apart.

Reading the CEP definition posted in your blog, it seems that we mostly differ on one aspect: what you call CEP covers most of what we call Decision Management. I see all sorts of technologies being involved in the Decision Automation world, CEP being the "event processing" part (and just that) and "business rules" being another one (and just that) and "business processes" being yet another one (and just that). All of those plus eventually predictive models (you mention neural nets and Bayesian networks for example in your analogy), optimization, etc. contribute to solving decision making challenges aka Decision Automation.

I personally prefer the term Decision Management to CEP for that ecosystem of technologies. The main reason is that we are not only processing events in there. The focus is purely on managing the decisions we make, whether they are reactive to streams of events or scheduled data processing or yet another trigger in another context. The focus is also on the management of those decision in the sense that business users want to own this logic (business rules or processes), etc. I feel that Decision Management for better or for worse is quite descriptive of the task at hand. It also emphasizes the management part rather than the processing part, and the management part is where the key business value is.

More food in the fire...

Carole-Ann Matignon
VP, Product Management at Fair Isaac

You are simply (and incorrectly, in my view) reducing CEP to low level event (sensor) processing, which was never the intent of the technology or the architecture.

I covered this in detail in my 8 part series, What is Complex Event Processing?

http://www.thecepblog.com/what-is-complex-event-processing/

and in particular, to analogy to the human body (in Part 1 of 8):

http://www.thecepblog.com/2007/05/14/what-is-complex-event-processing-part-1/

Hope this helps!

Your sincerely, Tim

The official statement from Oracle is here: http://www.oracle.com/us/corporate/press/017631_EN

As stated above, Haley is already used by Oracle Siebel products, and will continue to be used in that regard.

Oracle Fusion Middleware 11g will ship with Oracle Business Rules, and is the engine used by default in the decision component for BPEL processes.

I think people "get" the individual technologies but they don't necessarily get the vision. As a BRMS vendor, one may be tempted to redefine Decision Management as only BRMS (self-serving). Actually a couple of them did that in the panel.

Carlos talked about simplifying the technology stack, assigning roles to each one of the technologies, in a comment in another blog. I think he is exactly right. This divide-and-conquer approach to the complex problem of Decision Management is what we have been preaching. Decision Management is an approach to automate, improve and connect decisions. This is actionable. From the feedback I receive, people are getting it. We will get past the resistance of BRMS-only vendors!

Thanks,

Carole-Ann

You have a great point. I used to present EDM this way. I will think about putting together a new presentation and making it available via webinar.

I would welcome your input / feedback.

If you prefer to contact me directly, my email address is caroleannmatignon@fairisaac.com.

Thanks!

Carole-Ann

I am glad you mention the issue around lack of business focus in many of these standardization efforts. I agree with your point that business needs to drive what the focus of the standards should be and guide the efforts. Too many of these are just too technical and the results not usable by us your customers. It works for J2EE and the like because they are solving technical issues and were driven by technical folks. But for business rules, it's different.
In the name of us your customers: it's tough to get the time and the budget to participate. But I agree that we stand to lose the most.

I enjoyed your presentation. I actually am a customer of one of your competitors but I like the vision presented. It's very complete and pragmatic, I understand it, and it will help me position the work I am doing. What you could do in a future presentation is provide guidance for us on what parts of the vision are to be applied when in the life of the application. Maybe I missed that part.

Mark

I was surprising that the vendors "don't get it" about Decision Management. That is a "core" course in almost any school of business. Since a note pad can be DM tool so can anything else. Also surprising is that a lot of folks in the BRMS space have never heard of "Constraint-Based Programming." This, also, is an essential core curriculum for MBA. We were doing this with BASIC programs long before rules picked it up. As a matter of fact, every rule is nothing more than a constraint of some kind.

SDG
jco

Did you see the excellent article by George Soros in the New York Review of Books? Here is the link: http://www.nybooks.com/articles/22113

Soros borrows some of Minsky's ideas (without attribution, sadly) and coins his own term, "reflexivity" to describe how supposedly efficient markets act poorly and other actions support it. He even refers to the mortgage crisis as a bubble within a "super-bubble." Provocative piece, but not completely original.

Yes, I know Taleb's work. Too bad more people don't. I will take issue, though, with something you said. I still believe that the problem is that you can't see what you're not looking for. If you believe in market fundamentalism, you will likely miss these dysfunctional tendencies until it's too late, whether they involve new or mature mechanisms.

Neil Raden
twitter: nraden

There is a book by Nassim Nicholas Taleb, a former trader, called "Fooled by Randomness" that sheds some light on this. Before the credit crunch we've had a long period of expansion and very cheap money. Many of the analytics used worked remarkably well in this environment, but they had not been tested properly for different market conditions and under rare events. As the article I linked to above mentions this was in some cases intentional.

Adding to this many of the financial instruments were new and there was no real world experience on how they would behave when the economy soured, and of course it was not only the analysts who lied to themselves when testing their models, but also mortgage brokers trying to flip as many mortgages as possible. The GIGO (garbage in, garbage out) principle holds just as much true for analytics as for any computer program.

--
Aleksander

So why is Wall Street melting down?

Clearly, analytics aren't everything. Our financial system is pretty complicated and subject to abuse and fraud. The current crisis is aligned with the greed of the mortgage brokers and the mortgage bankers. Once in a while the financial press will point the finger at the hedge funds and investment brokers that shoved mortgage-backed securities down the throats of other investors.

Hmmm.

Wasn't anyone watching this? After all, interest rates started to creep up a few years ago, the economy started to turn down, default rates started to appear at around the same time. Is it possible that the quants were so buried under leveraged layers of derivatives and other exotic instruments that they didn't see the coming storm? This seems like a pretty big movement to miss. After all, if you're sitting on top of a few hundred billion in debt that is on the razor's edge of liquidity, wouldn't you spend some time looking at it more closely, especially with such broad macroeconomic factors staring you in the face?

Maybe the problem was just that — too much attention to the monetary and business-related factors and not enough attention to the movement of markets on a broader scale.

In the early '70s, I had the unique opportunity to take economics classes from the legendary (but until recently, obscure) Hy Minsky. Minsky is known for the "financial instability hypothesis," which proposes economic expansions become unsustainable booms ending in crisis and economic unraveling. Speculation. Greed. Disaster. I first heard the phrase "Chaos Theory" from Minsky thirty-five years ago.

Minsky has suddenly become very popular (unfortunately he passed away in 1996). One of his memorable quotes in class (there were many) was: "All panics, manias and crises of a financial nature have their roots in an abuse of credit." He used the Dutch Tulip mania of the 1600s as an example. He believed that financial systems experience rounds of speculation that, if they are severe, end in crises. Minsky was considered a radical for his stress on their tendency toward excess and upheaval.

Minsky showed that bubbles are an inevitable result of market activity. Buyers who show gains with a successful strategy encourage other buyers until it stops working. When investors have to empty their portfolios of even their prime holdings to cover their positions, markets start to circle the drain. At that point, the "Minsky moment" is obvious.

And it's here — Bear Stearns, Countrywide, Lehman Brothers, Merrill Lynch.

So the question is, why didn't the best and the brightest see it coming? We need to do some soul-searching. Is there really any benefit to advanced analytics and an analytics culture if doesn't see the train coming through the tunnel? Or is it something else? I think that no one wants to believe a "Minsky Moment" is coming.

You only see what you're looking for.

-Neil Raden
twitter: nraden

They do it by charging up front for x number of miles. You tell them what your odometer is currently and you are covered from your current odometer reading to your current odometer reading plus x. Pretty simple really. The cool part is that they don't have to track people or verify they are telling the truth. They simply check to see if the odometer reading in within the distance insured if you get into an accident.

I know all this because I'm a share holder and former employee. So don't just take my word for it, check them out. The less you drive the less you pay.

-Jeff

http://bits.blogs.nytimes.com/2008/09/18/how-wall-streets-quants-lied-to-their-computers/

We probably need a really statistics-savvy governance function to stop such things.

--
Aleksander