Complex event processing boosts agility to create competitive advantage

Companies today hope to gain a competitive edge based on their ability to respond quickly to threats and opportunities. That edge can come from highly responsive, near-real-time systems based on complex event processing (CEP) techniques used for financial applications on Wall Street. Now these CEP techniques are being used more widely in enterprise applications on Main Street.

Examples abound:
--Fraud analysts are alerted when the same credit-card number is used in several cities almost simultaneously.
--Power-line fault histories and locations can be quickly compared against real-time weather feeds to alert utility maintenance teams about likely trouble.
--A healthcare provider can rapidly check a type of test regimen against a patient's insurance coverage.

CEP seems to be a way to greater corporate innovation. For many companies, the move to event processing means a shift in perspective on software architecture.

Complex events have unique characteristics, and people involved in the field still argue about the definition of various terms that describe CEP. "Event-driven architecture" is sometimes used to express the way in which CEP problems are handled.

While the inner workings of various CEP engines differ in composition, their structure may not particularly concern application development managers, who will instead tend to focus more on how a CEP engine is programmed. "If you talk to users of CEP, they may well be interested primarily in what they can do with the user interface to the engine," says David Luckham, emeritus professor of electrical engineering at Stanford University and author of Event Processing for Business: Organizing the Real-Time Enterprise (John Wiley & Sons, 2011).

To ease programming, CEP vendors support SQL-like, visual and related programming techniques. These fit well with existing programming skill sets. Sometimes CEP products are distinguished based on whether they support query-based CEP or rule-based CEP. But tooling is where the development team meets CEP head on.

The varied composition of the "databases" underlying different CEP engines is in part due to their varied lineages. Some hail from the world of rules engines, some from business activity monitoring (BAM) and some from other backgrounds.

Event-driven systems may show kinship to relational databases or middleware messaging systems, or to hybrids that merge both schools. Sometimes, but not always, underlying databases are described as "streaming databases" or "event streams.” Vendors with CEP-related products include Espertech, IBM, Informatica, Microsoft, Oracle, Progress Software, RedHat, StreamBase, Sybase, Tibco and Vitria, among others.

In most CEP engines, data records are processed before they are stored. That makes it different than, say, data warehousing. CEP differs from the conventional RDBM in that a small amount of data may be matched against a large number of queries, rather than a small number of queries being run against a large amount of data.

Events—simple or complex—can be analyzed as they "stream" through operating systems. There’s a special emphasis on handling time-series data and establishing time windows that show event data. Within these windows, new data are compared with known event patterns.

Complex events are collections of several simple observable events that are of interest. To cite a common example, you might want to see whether the same credit card was used simultaneously in two locations, a situation that could indicate fraud.

Observers say some CEP engines tend to exhibit more affinity with conventional relational databases, while other CEP engines exhibit less such affinity. But from a programmer's point of view, the differences may be narrowing.

"Some CEP products look more like DBMSes than others. But the differences are a lot less than they used to be," says W. Roy Schulte, a vice president and distinguished analyst for Gartner Inc. "Five years ago, there was a big difference." One driver for convergence is tooling that supports SQL-like means for dealing with the unique characteristics of CEP data streams.

Staying with the familiar has benefits. But the nature of event-driven architecture is such that some CTOs and development managers will consider new programming models. This can be a difficult decision point. You may not want to change your programming style, but a new class of applications may actually best be handled by a new programming style, suggests Curt Monash, president of the Monash Research consultancy.

"There is a programming model that revolves around events, and taking action on them, which is different [than conventional data base management systems]," he says.

Where speed or complexity is the issue, interest in "how the engine works" may become more pointed. Here, most CEP engines vastly outpace general-purpose servers.

"The products are all optimized for performance in very demanding situations. In some cases, one is better than another. They also have rather different programming models,” Monash says. "There are ones that are stream based that emphasize visual modeling. These and others let you write code if you insist. There are rule-based and language-based [programming models]. Each [product] has its clear roots or emphasis. And that is often a very natural basis for choosing between them.”

One may create a SQL-like query language with special capabilities to handle streaming data and continuous processing. Another may provide a graphical user interface for modeling, allowing team members to drag and drop event components to compose applications. Yet another may provide a graphical event-flow language.

Meanwhile, as with other upstart technologies, CEP may ultimately have difficulty finding a steady job in the mainstream--that is, if incumbent database and business intelligence technologies keep up. "CEP is a contender," says Monash. "But it may not win."

CEP is an alternative to database technology. Outside a couple of niches, it’s not clear that such an alternative is needed, given how capable and varied database technology is.

Where do the workings of the underlying CEP data-handler truly make a difference? One place to look: especially complex event architectures requiring high speed and high-volume processing.

"Some of the problems that arise in applying CEP involve the complexity of the event patterns being used," Luckham says. "If you have to check several events and a state reference representing—say, the history of events in the past hour—then you have to both detect the pattern and see if the state is true," he says.

That task becomes a bigger issue when fragments of patterns are involved, he continues: "You often get partial instances of patterns. You may have several or many partially matched patterns. What do you do with that in CEP? When you get into those types of problems, at that point the structure of the [underlying event database] becomes of the interest to the application specialist.”

The issue could well depend on the kind of problem you’re facing, he says. For example, if you’re trying to do arbitrage trading at several thousand events per second on several stock market feeds simultaneously, the way the CEP engine handles events will certainly make a difference.

There can be plenty of overlap between EDA and service-oriented architecture (SOA), just as there is with synchronous and asynchronous systems, according to Kim Kazmaier, a technical fellow for Bank of America.

Kazmaier, who has helped to lead a multiyear SOA effort at Bank of America's consumer division, gave a presentation on the effort at a recent Gartner Inc. conference. The effort is now beginning to incorporate EDA and CEP. The early efforts to do EDA build off of messaging middleware expertise his group has already amassed, he says.

"Most organizations are now trying to look at capturing the value of their transactional experience," he said. "You can view it from an opportunity as well as a risk-reduction [point of view]."

Users should anticipate faster operations with CEP: "Event engines run four or five times faster than a typical SOA [engine]," he said. But that doesn’t mean you automatically achieve the speed you want.

"One thing I have learned about CEP is: If you mix the workloads too much in a single engine, you might not get your [desired] throughput," Kazmaier warned. As a result, he called the use of multistage design patterns "a good idea."

READER FEEDBACK: Are you using complex event processing for innovation? Or for faster, more efficient operations? Or both? If so, what three tips could you offer to others considering the same path? Please e-mail Site Editor Anne Stuart at

About the Author

Based in Newton, Mass., Jack Vaughan is editor of chief of ebizQ's sister site, Contact him at

More by Jack Vaughan