Complex event processing: An emerging necessity for today's businesses

Editor’s Note: In this Q & A, ebizQ’s Peter Schooff talks with David Luckham about trends in complex event processing (CEP). Luckham, emeritus professor of electrical engineering at Stanford University, is most recently the author of “Event Processing for Business: Organizing the Real-Time Enterprise” (Wiley, 2011). Their discussion, excerpted from a longer podcast, has been edited for clarity, length and editorial style.

ebizQ: Where exactly are we today with complex event processing?
CEP is being used by many organizations in their business operations and also by government departments on a worldwide basis. It is being applied in different kinds of commercial applications that process real-time event feeds. Quite often, CEP as a technology is under the hood and you don't even see it at the user interface of the applications. I will describe some examples during our conversation. Stock pricing is well known, but there are lots of others.

ebizQ: Let's say a company really hasn't engaged with CEP. Why should they consider it now?
That really depends on the company's business, exactly what sort of business they’re in. But a fact of life is that most businesses are becoming more and more dependent upon the Internet and upon the use of available data in real time. A phrase that I like to use is “right-now time.” A key requirement here is processing the data as it arrives at the company.

So I would say that any company that has made rapid responses right now to incoming events will eventually need an event-processing capability that uses CEP. One other thing I might say here is that when a company comes to that conclusion, probably by analyzing its business and also the competition, they will get into a “build-it-or-buy-it” discussion. The company will then need to analyze what kinds of CEP processing is provided by the commercial products on the market and whether that capability will meet their needs.

ebizQ: Obviously, quite a lot of events occur every day in business—some very important, some not important at all. What events would you say are most important to the business's bottom line?
Again, that depends entirely upon the business. Sorting out the relevant from the irrelevant events is the first and most important aspect of event processing. I would say at very least that a company has to pay attention to its own internal events. If you don't know what your own company is doing in real time, then you're probably in trouble. For example, [you need to address questions such as] “Is the inventory department keeping up with the sales department?” There are always questions about maintaining company secrets and the possibility that you might have spyware in your IT. So certainly a company needs to pay attention to its own internal events.

Beyond that, there are gazillions of external events coming from every quarter of this planet—and outer space, I might add. That, basically, is the Internet world that we now live in. And what events are actually important depends on a whole host of factors.

But I'll say two things about that. First, the importance of an event as it happens depends upon what else is happening. Sometimes an event can lose importance or gain importance very quickly. It’s not a constant attribute. [Second,] business intelligence is all about sets of events and their timing, not necessarily about single events. This is why event processing involving CEP is important, because it enables a company to build capabilities that detect and process sets of events that it needs to know about.

ebizQ: How would you say CEP relates to BPM?
BPM has been an area of business operations for a long time. What’s happening [now] is that CEP is becoming an enabling technology in real-time business processes.

Adopting BPM is not an easy undertaking for a company. It can go on for years as one figures out what processes one needs and where they should operate, and how to prioritize them. And CEP plays a role in the underlying technology that drives BPM—when to trigger a process, how to prioritize which processes can run and so on. So you'll find CEP built into business processes more and more today.

ebizQ: Which industries are seeing the greatest benefit from event processing today?
Financial services, right off the top, because they were early adopters and big spenders. There are some event-processing vendors that specialize in the financial-services market. But there are many aspects to consider [starting with] how you're going to use event processing; there are defensive uses to prevent losses, versus positive uses to expand the business.

As everything becomes more of a right-now business, it turns out that event processing is an essential technology in many [other] industries. A second sector is the military sector, which is a very large user of event-processing technology in everything from supply chain management to battlefield logistics. [Others include] the security and fraud detection business, and transportation, which includes airlines, trucking, railways, shipping—container shipping especially.

Then you’ve got the energy sector, which goes everywhere from smart grids— I'm told there are over 100 smart-grid projects worldwide at the moment—down to the retail end of the energy operation, where the utilities are selling smart meters to the household. Then there are the areas of government, homeland security and intelligence. And here, of course, we might get into identity theft and so on. So there are many, many areas where industry is seeing a benefit from event processing.

ebizQ: In reading your book ["Event Processing for Business"], which is excellent, I came across a phrase you used: “a completely holistic event-processing system.” Can you explain that a bit more?
The rise of holistic event processing systems is a big question now. You can see this happening in some businesses. [For instance,] banks are expanding their areas of operations into, say, the travel market and shipping. What happens here is that the [number of] types of events that need to be processed expands. Instead of just dealing with stock market events or customer account events, you’re now dealing with travel events and weather events and demographics and so on.

So holistic event processing refers to systems that process very large numbers of types of events. Now this is different from the numbers of events themselves. You can have thousands of events per second being processed by a stock trader, but he's processing just one or two types of events. So I'm looking now at systems that are not only processing lots of events, but lots of different types of events. Those systems I call “holistic events processing systems.”

[For example,] air traffic operations are becoming more unified. If you look at the North Atlantic, you've got unified air traffic operations on a very large scale between the U.S. and Europe. This involves not only aircraft guidance, but all of the events that lead up to the scheduling of an aircraft.

So the number of different types of events is expanding. If you look at the event processing system that's evolved involved in unified air traffic operations across the Atlantic as a whole, it is processing a very large number of different types of events. It’s built up from lots of little systems, smaller systems.

There are a number of things that I would say about these kinds of event processing systems: They’re happening now, but we’ll recognize them say ten years from now. They're not planned; they’re just happening in a chaotic, evolutionary way. And they have to happen due to technical, political, and social forces. That's why they're going to happen. It’s a matter of keeping costs down; it’s a matter of being able to answer lots of different questions that require lots of different kinds of event inputs.

[Another example:] In the medical field, pandemic-watch systems can be used to recognize the early emergence of a pandemic like bird flu. Through world environment-monitoring, we're beginning to become conscious of the fact that the world's resources are very limited and that we’re eating them up. Monitoring the use of those resources on a large scale across the globe [would involve using] a holistic event processing system involving many different aspects. [Other examples include] global weather-forecasting systems and government-to-military intelligence.

ebizQ: Looking ahead, what do you see?
If you look at things like the traffic control systems for large metropolitan areas, they're becoming more automated. There's a very good possibility that in the future, we won't drive our cars in heavy traffic situations. They’ll drive themselves and they'll communicate with the cars around them and with the traffic system. That will involve an event processing system that is holistic, that is built up of lots of smaller event-processing systems, and its purpose is to get the drivers to work or wherever they’re going in a reasonable amount of time.

The whole area of managing our information itself will probably cause the evolution of holistic event processing systems for preventing identity theft and protecting the individual—but probably also for spying on the individual as well. There are always two sides to these kinds of technologies.

But I would say that one can see the precursors of holistic event processing systems developing now as we speak. Basically, an inescapable proof is that we have become an event processing society and there's no going back: Holistic systems will be in our future.

READER FEEDBACK: Is your organization using CEP? Do you have experience with the kind of holistic event processing systems that David Luckham describes? ebizQ’s editors would like to hear your thoughts. Contact Site Editor Anne Stuart at

About the Author

Peter Schooff is a former contributing editor for ebizQ, where he also managed the ebizQ Forum for several years. Previously, Peter managed the database operations for a major cigar company, served as writer/editor of an early Internet entertainment site and developed a computer accounting system for several retail stores. Peter can be reached at

More by Peter Schooff

About ebizQ

ebizQ is the insider’s guide to next-generation business process management. We offer a growing collection of independent editorial articles on BPM trends, issues, challenges and solutions, all targeted to business and IT BPM professionals.

We cover BPM standards, governance, technology and continuous process improvement, as well as process discovery, modeling, simulation and optimization, among many other areas. We follow case management, decision management, business rules management, operational intelligence, complex event processing and other related topics. We closely track important trends such as the rise of social BPM, mobile BPM and BPM in the cloud. We also explore BPM’s use in functional areas, such as supply chain and customer management, and in key verticals, such as financial services, health care, insurance and government.

ebizQ's other BPM-oriented content includes podcasts, webcasts, webinars, white papers, a variety of expert blogs, a lively online forum and much more.