Complex Event Processing (CEP) Blog

The Personalisation of Event Processing

Paul Vincent — Thu, 18 Mar 2010 20:31:20 +0000

I was struck by Jim Sinur’s latest post about how far “customer control” in B2C relationships has yet to run. Jim talks about how a bank decided to change its policy on overdrafts from:

accepting payment events on overdrawn accounts, hence increasing charges

to:

refusing payment events that would make an account overdrawn

This struck me as odd. Surely, if banks have controls like this (and they do), then they can have account settings to allow either of these options, to suit either the individual (or the bank!).

Better still, they could allow the adjustment of event controls to suit the individual and their needs. This is what personalisation is about. I should be able to specify whether the bank lets me go overdrawn for payment events based on payee, timing, and so forth… The bank might want to monitor my income spending profile and make suggestions (or specify defaults) to account settings and behaviors. For example if it detects my income changes then it might suggest, or set defaults to, rules / service levels that have suited similar customers.

Compare and contrast to current bank accounts, where occasional (but seemingly regular) marketing initiatives “invent” new banking “packages” providing certain services and entitlements (such as low cost insurance, discounted overdraft facility, credit balance interest rates etc).

Bank IT departments might complain however that their mainframe batch operations cannot possible handle individual business rules for each user. To do that they might need to move to… event-by-event processing, using something that can join customer parameters with customer transaction events… like an event-driven rule engine, perhaps? Possibly some banks are already moving in this “event-driven process” direction…

Why Agent Technology for event processing? by James Odell, CSC

Paul Vincent — Tue, 16 Mar 2010 11:21:11 +0000

Over the next few weeks James Odell, known for his work on OO, UML and agents, will be offering some thoughts on agent technologies on this CEP Blog. Over to Jim…

The notion of event processing agents (EPAs) is quickly becoming a topic that more and more organizations are considering. We’ve been building systems that employ event processing without agents for quite some time. So, why agents and why now?

One of the primary reasons is that we are no longer in the era of mainframe computing, when both companies and applications were typically command-and-control oriented and organized in vertical silos. With the combination of the Internet, fiber optics, and PCs, the business and technology playing field has been flattened. No longer primarily top down, it has changed to more side by side-as individuals, small groups, and organizations interact around the world. In other words, the events around us are based on complex web of activity that is less command and control and more horizontal connecting, collaborating, and competing.

When the events among requesters and providers are few and change is not an issue, agents are not needed and conventional IT techniques can be employed. However, when the links require complex, dynamic binding and are subject to rapid change, agent-based approaches should be considered. Prior to the demand for global collaboration, we developed a centralized controller to ensure that all interactions would be appropriately managed. Now, this is no longer possible. The world is too big and a central bottleneck would paralyze the effort. As a result, the access techniques now require a more horizontal style of interaction-rather than one that is centrally administered.

Agent technology is a primary enabler to support this new direction. In fact, without agent technology, our current technology will not scale to support the ever-increasing global interaction. More importantly, it will enable us to create and support a whole class of IT applications and approaches that we previously could not have developed-this especially includes complex event processing.

In the next installment, I will address the question: “What is the relationship between agents and events?”

The Return of the Expert System?

Paul Vincent — Fri, 12 Mar 2010 17:37:18 +0000

There was an interesting discussion on LinkedIn recently about Expert Systems versus Business Rules (and, my contribution, versus CEP). This was posted on the BPTrends discussion board by business rules guru Ron Ross - probably not coincidentally as not only are business rules and decisions related to business processes, of course, but also one of the better business books on Expert Systems in the 1980s was by one Paul Harmon (presumably the one and same Paul Harmon who runs BPTrends… ).

Expert systems were in-vogue in the 1980’s as decision support systems. The term was most popular in the US, as I recall, whereas the Europeans preferred the more defensible term “knowledge based systems”. Nonetheless, expert systems were usually given a set of criteria, not least of which was the first:

expert levels of performance (comparable to a human expert)
ability to explain its reasoning
ability to learn new knowledge.

Technologically, these tended to use features like CBR , or inference rule engines that were backward-chaining (goal-driven) - they tried to prove relevant hypotheses through questions and answers. Usually they were connected only to a console to support interactive “conversations” with end-users.

Such decision-support systems were ideal for technical support call centers, where the operators interacted with the expert system on behalf of callers. This also had the side-effect of training the operators. Indeed, one call center (if I recall correctly) found that after some months use, all its expert system operators were qualifiable as “experts” in the specialist domain they were dealing with!

Now fast forward 2+ decades, and we find ourselves dealing with vast amounts of information and events, automating and optimising business systems to save costs and improve performance. Here we see parallels with between expert systems and CEP implementations:

best available expert knowledge encoded from the available Subject Matter Experts
take expert event-driven decisions (and, if needed, log the rules used for monitoring offline or in a dashboard)
use rules to determine whether goal states are achieved…

This is a long way from the interactive expert systems of old, but nonetheless shows similarities in both use cases (e.g. providing expert decision-making) and technologies (e.g. TIBCO BusinessEvents uses at its core an inference rule engine of the type used in many such expert systems).

I don’t advocate resurrecting the term “expert system”, even though I see an ex-colleague sports the job title of “Director of Expert Systems” at a well-known media company! But “expert performance” - obtaining results equivalent or better to what a human expert would achieve - is still something to strive for.

Events versus Data

Paul Vincent — Fri, 12 Mar 2010 11:40:43 +0000

A recent question came up regarding the difference between “events” and “data”. After all, to a DBA, an “event” is just another data record but with a timestamp! Of course, that is the “static” view of an event - useful only for archiving and “post (-event) processing” activities like data mining, BI and analytics.

However, a more technical answer is that data is derived from events. Customer data? Derived from customer registrations and associated checks - all events. Amount owed? Derived from purchase events. And so forth. So a better answer for the DBA is that “data” is just “information collected from, and associated with, prior events“.

In Complex Event Processing, of course, we are processing this “event data” as it occurs (within the limitations of our system latency, of course!). Typically this event data is correlated with previous events which can be either “relatively recent” (and considered as stored events or event objects) or “historic” (and considered as traditional data), although these are necessarily imprecise concepts. Access to historic data is typically carried out in CEP tools through good-old-database-services: calling the operational data store associated with legacy systems, for example. Another approach is to “event-ise” the legacy applications, extracting new events from logs and data sources as they occur.

TIBCO examples of the above approaches to data access in event processing are TIBCO BusinessEvents’ database interface (what we call DBConcepts) and TIBCO Adapters (e.g. for ERP and HR systems).

Notes:

The EPTS glossary does not define “data” but does mention that event objects contain data…

Location awareness and Twitter

Paul Vincent — Wed, 10 Mar 2010 19:20:49 +0000

Thanks to TIBCO’s Patrick Sapinski for some news on a CEP-driven Twitter application (albeit not public yet) that used location awareness principles to “tweet” people depending on where they were (e.g. passing a certain location). The Twitter side of the story was apparently the easy part. Patrick writes:

“We did not do any Java coding to accomplish this. We just used the TIBCO BusinessWorks HTTP activity to send tweets to Twitter. It was pretty easy once I configured the messages, URL, etc.”

Of course, for corporate applications / private tweets there are also services like Tibbr, which I signed up for this week… meanwhile I’m happy sticking with iGoogle to monitor “the real thing” via RSS feeds!

Up-Scale Your Apps with Distributed Caching

Paul Vincent — Wed, 10 Mar 2010 18:45:53 +0000

… was the subject of today’s Forrester-IBM webinar on distributed cache technology, with both Forrester and IBM citing CEP and EDA as users for this technology, amongst others. The overriding driver for this tech being eXtreme Transaction Processing, which we might just refactor as eXtreme Event Processing for the purposes of this blog!

One minor quibble: John Rymer of Forrester did the introduction and during so classified the cache market as .NET, Java and NoSQL camps, with TIBCO placed in the Java camp. This might seem a fair classification of a complex market area, but of the 2 relevant TIBCO distributed cache offerings:

TIBCO BusinessEvents, although Java-based, is more accurately described as a CEP product that embeds a distributed cache - it wouldn’t normally appear on a vendor list of distributed cache technologies;
TIBCO ActiveSpaces is more accurately described as a data grid, but has .NET, C and Java interfaces. I’m sure other caching / data grid products have similar multiple interfaces - after all the client is just “an interface” to the cache / grid.

Spare a thought in passing, though, for the OODBMS guys. Amongst this buzz about data grids and caching, I notice the Forrester blog is reporting that the Progress guys (disclosure: a TIBCO competitor in some areas) are now considering their ObjectStore OODBMS a “legacy platform”. Plus ça change, perhaps.

How does CEP fit into BPM and SOA environments?

Paul Vincent — Fri, 05 Mar 2010 00:47:39 +0000

There were some customer discussions recently on the conceptual relationships between the CEP and BPM and SOA “software stacks”. This coincided with the announcement of the OMG-backed Event Processing Consortium being set up (alongside the merger of the SOA and BPM consortia), events which themselves can be interpreted as that event processing has some special role to play alongside BPM and SOA.

[Disclosure: note TIBCO is not a member of either the OMG's EP or BPMSOA consortia - they seem to be focused on end-user rather than vendor participation - but is an OMG member and participant in standards development. We currently see the EPTS as the main advocacy group across vendors, academics, analysts and end-users, but will be monitoring the progress of the EP Consortium].

So here are a few simplistic patterns on how CEP (event processing) relates to BPM (processes) and SOA (services)…

CEP Pattern	Example	Diagram
1a. Standalone CEP This is a bit of a misnomer - you don’t identify event patterns without some intent to use them, and at the very least such a use would be in a BAM type role displaying interesting correlations for some business person - who of course is engaged in some kind of business process, which may or may not be managed by BPM…	Monitor a production process to provide an “additional view” or dashboard for the process control manager.
1b. CEP enriching BPM processes and/or SOA services. This is the conventional view of CEP, detecting the complex events that are of interest to, and useful in, appropriate processes and services. Complex events in these cases can be as straightforward as deducing that a deliverable has been completed, or some process truly initiated. Typically the CEP system is transforming source events into business events, for onward use in (the) business processes.	Identifying exception events in a business that need to handed to a workflow or case management system for resolution.
2a. CEP monitoring processes and services This is where the sources of events are the managed processes and services themselves. This process and service monitoring is used to detect exceptions, disparities across systems, and system performance… Note that effectively this pattern is a combination of patterns 1a and 1b above.	Detect when response times exceed some metrics and suggest corrective actions such as reallocating resources.
2b. CEP-based decisions for processes and services This is where I need to make intelligent decisions for the process and service layers, using the CEP layer as a monitoring, shared decision management component. Note that effectively this is a slight extension of pattern 2a above.	A BPMN “rule activity” sends a decision request to the CEP engine to get a valid decision for a process decision point; the CEP engine monitors the decisions made.
3a. Dynamic process and service control This is where the events from processes and services, and external services, are combined to select which processes and services are relevant to use for the current context. In effect, the CEP engine becomes the controlling agent for the business processes and service engine, handling for example dynamic process selection. Note that this pattern is a further evolution of patterns 2a and 2b.	In a complex business process for ever-changing fulfillment problems, CEP-based rules determine which sub-processes are valid based on incoming information.
3b. Embedded processes and services within CEP The final evolution of the above is when you argue that the functions of the BPM and SOA stacks can be subsumed into the CEP layer. In reality this is usually only a partial subsumption, as otherwise the centralization of services into just 1 layer could be perceived as contrary to the very idea of SOA! So this covers things like event-based policy implementations being embedded as CEP rules rather than as external services, but alongside some external services such as an operational database. Indeed, one could argue that in this case the CEP event processing agents are themselves really part of the SOA layer, not the other way round!	A service gateway controlling access to existing services, but embedding decisions, service policies, and business rules.

So, did I miss anything?

Complex Event Processing: a technology evaluation check-list

Paul Vincent — Thu, 04 Mar 2010 19:43:54 +0000

One of the challenges for organisations investigating event processing technologies is “how do we know what is important”? Obviously everyone will have different views on this. Nonetheless, as TIBCO has the most experience in CEP across multiple domains and the largest customer base for a CEP technology tool, it seems to make sense to present a sample evaluation guide that shows the typical considerations and features needed.

First, a few comments. TIBCO deals with enterprises who can have sophisticated requirements around their complex event processing needs. TIBCO customers generally use CEP for operational intelligence applications covering anything from business activity and event monitoring to automated intelligent business processes. Hence TIBCO solutions handle high throughput (not just latency) and payload size (business messages can get very large), whilst providing the fault tolerance and resilience capabilities that 24×7 organizations require. But every class of CEP application has different requirements: for example an algorithmic trading solution might prioritise ultra-low-latency over other features like failover-support and logic and payload complexity. Ergo, for every use case the “example scoring factor” provided below will need to be adjusted to match the feature’s importance to the use case.

Feature	Detail	Reason	Example Scoring Factor
1. Event Channels	A. Specific channel types required	Example: JMS, MQ-Series, HTTP…	10
	B. Custom channels	Example: custom event type for SCADA	3
2. Modelling	A. Event models	Event metadata, hierarchy, inheritance	10
	B. Event object / concept models	Object history, hierarchy, inheritance	10
	C. State and status of complex events	State model and flow	8
3. Processing Elements and Expressions	A. EventConditionAction rules	Basic event filters, aggregations	10
	B. Inference rules	Intelligent event processing	6
	C. Continuous queries	Event streaming aggregations, statistics	6
	D. Temporal regular expressions / logic	Concise pattern recognition	6
	E. Specialist analytics and statistical functions	Capabilities for pattern discovery, possibly in conjunction with developer interface (see 7)	6
	F. Other context support	How to handle context across time, such as GVs etc	6
	G. Extensibility	How to add custom functions / algorithms / analytics?	6
4. Event augmentation	A. Database access	JDBC read/update etc	4
	B. SOA access	Invocation of existing services	6
	C. BPM / workflow access	Invocation of manual steps	3
5. Stateful processing and management	A. Event storage in-process	In-memory support	8
	B. Event storage out-of-process	Low latency cache and/or high latency backing store	8
6. Platform capabilities	A. Scalability via distributed processing agents	For large processing loads (logic and/or event size and/or rate)	8
	B. Scalability via distributed event cache	For supporting event patterns over processes, resilience	8
	C. Failover / resilience features	Hot standby and event handling if agent fails	8
	D. Hot deploy / event-based updates	Hot deploy of new rules, queries or patterns, or dynamic update through events	8
	E. Operations and Management	UI and/or reports to monitor performance and/or adapt capabilities	8
	F. Performance supporting features	Components to minimise latency like preprocessor, algorithms etc	10
	G. Deployment environment support	Desired or preferred hardware/OS/cloud platforms	8
7. Development Environment	A. Standardised GUI tool	Example: Eclipse-based	8
	B. Verification tools	Logic verification such as diagram generation and reports	6
	C. Validation tools	Unit and bulk test capabilities	6
	D. Debugging	Local / remote debugging capabilities	6
	E. Team features	SCCS / project repository / team development	4
8. Business interface	A. Business Event / Rule / Decision Management	Management (editors, impact analysis) for business-controlled event processing aspects	6
	B. User Workflow / control	Control and security for business updates to event processing	6
	C. Event monitoring dashboard	Display of events, and complex events such as KPIs	6
9. Project considerations	A. Track record	Numbers of Customers / users in same domain	6
	B. Availability of SIs and development partners	Available assistance if needed	2

One final checklist or scorecard item cannot be added to the list above, but is at least as important as the technology factors. This is the client-vendor relationship, and I was reminded of this when another CEP vendor tweeted recently about how they had been gaining business partly due to another vendor’s “aggressiveness” in business dealings. So, just like everywhere, caveat emptor…. and maybe add a higher score to 9.A. above!

As usual, comments, feedback and suggestions welcome.

Analytics moving to Real-time via CEP?

Paul Vincent — Tue, 02 Mar 2010 13:33:29 +0000

Two events last month showed indicators of a convergence between the analytics world and CEP world.

Firstly Louis Bajuk-Yorgan from TIBCO Spotfire attended the Predictive Analytics World conference in San Francisco. He reported that:

Three key themes showed up multiple times throughout the talks-the growing importance of text mining, the application of net lift modeling to determine the real results of a marketing campaign (ignoring those people who would have bought anyway), and (most interesting to me) the importance of operationalizing predictive analytics.

In his opening keynote speech, Eric Siegel (the conference chair) saw the most important innovation in the field of Predictive Analytics focused on applying predictive analytics to operational decisions (as opposed to more established application areas such as customer churn & product recommendations). In a later talk, James Taylor of Decision Management Solutions (and co-author of the great book “Smart (Enough) Systems”), echoed Eric’s emphasis on operational results, encapsulated in the phrase “Action support, not just decision support.” James advised building an analytic platform that focused on the end game: the need to operationalize analytic decisions.

This is great validation for us, since operationalizing analytics is at the heart of TIBCO’s vision for its combined platform with Spotfire and S+ (as shown in products like Operations Analytics).

Then a week later Andreas Gerst from the TIBCO BusinessEvents team presented at cepconf in Munich, Germany. Andreas presented on CEP and Data Mining, and in particular how both these complement each other for advanced operational intelligence around customer management. Andreas used TIBCO BusinessEvents and TIBCO Spotfire Miner as his example technologies, mentioning techniques like PMML for moving from analytics to real-time event processing technologies.

Processes = event-driven decisions and actions?

Paul Vincent — Wed, 24 Feb 2010 21:08:03 +0000

In a previous post we commented on IBM’s Conceptual Model for Event Processing Systems, but failed to observe that IBM’s rule engine group, Ilog, was noticably absent from its contributor list. So perhaps it was both fortuitous and timely that Ilog’s Daniel Selman recently added another “viewpoint” on the position of event processing (aka the rules viewpoint).

Daniel’s take on the “primary users (sic) of rules technology” are the (use cases for) automating decisions, event responses, processes and inferences. I think these might be better classified by renaming then as decision processing, event processing, and business process processing. But inferencing, which Daniel notes as being a technology to support Artificial Intelligence, is not so much a (user or) use case, but a means of providing knowledge-based reasoning to support any of decisions, processes, etc - “AI” is not (or should not be) a means unto itself.

Standard Event Processing Design Pattern

Indeed, one could probably argue that:

all business processes are driven by events of some kind, and involve decisions and (re)actions;
all processes, decisions and event processing are context (i.e. state) driven; however, some processes and algorithms are used in a subordinate fashion to a stateful process (consider a typical decision service that is subordinate to its application server and database layers);
inferencing can be used to enhance any part of the event-handling process as a form of declarative rule control.

In TIBCO’s experience, an event driven rule engine (like TIBCO BusinessEvents) can be used to provide dynamic business processes, event-driven decisions, and real-time control mechanisms - and often all 3 - exploiting and building on the fundamentals of complex event processing.

So, on the utility of rule engines, we totally agree with Daniel!