Editor’s Note: In this Q & A, Forrester Research Principal Analyst Mike Gualtieri speaks with ebizQ’s Peter Schooff about the growing convergence of complex event processing (CEP) and business activity monitoring (BAM). Their conversation, excerpted from an accompanying podcast, has been edited for length, clarity and editorial style.
ebizQ: From my perspective, there's been a lot of talk about both complex event processing and business activity monitoring and business activity monitoring. How do these two technologies complement each other?
I think you're right to say that they complement each other because business activity monitoring
is about monitoring the business, just like it sounds. Complex event processing
is a lower-level technology that can support that.
So imagine you're trying to monitor business activity like transactions going through an e-commerce system or deliveries if you're a large logistics company. There are certain events that may be notable in those business event streams, [things] that are happening that you want to take action on. CEP is an underlying technology that can--in some cases but not all-- help with detecting those events.
ebizQ: So when should company should adopt complex event processing, and when should a company adopt business activity monitoring?
Let me answer that, but let me take a step back on the complex event processing.
Complex event processing is a relatively new and very complicated technology because it does temporal pattern recognition at very low latencies. That sounds very complicated, but basically, what it means is it can detect if two things happen very, very close together.
The initial market for CEP engines was none other than Wall Street because you’ve got an enormous flow of information. So imagine Microsoft’s [stock] price goes in at $27.50 and then IBM goes in at $90 within 3 milliseconds. That's something that CEP can detect and, of course, then you might want to take some action on that. So that is an extreme case [but] that’s CEP's sweet spot and there's very, very low latency transactions.
In business activity monitoring, you may have--and you often do have--similar circumstances. It may not be in the millisecond level, but you'll have a sequence of events that alone don't tell you anything. But together, it's a pattern that you then want to take some action on to avoid a missed delivery or keep a manufacturing line running, for example.
ebizQ: What are some the internal barriers to adopting either technology?
It’s always the problem with adopting any new technologies that [doing so] often requires connections to a lot of data sources. When you have very large organizations that have a portfolio of 300 applications and a very heterogeneous infrastructure, just connecting to and capturing those data sources is a key challenge--and the first challenge that you have to deal with in adopting these technologies.
The second challenge is to really identify what patterns actually matter, because there's a lot of noise. You can imagine all the data running through; some of it is not relevant, some of it’s noise. You have to find the signal in that data. So some analysis work has to be done by a combination of business and technical people to figure out what matters, [answering questions such as] “What do we actually want to detect? What are those patterns we want to detect?” That can be a lot of work.
But the good thing about business activity monitoring is you can start small. You can just focus on “if we could just know this one thing, then we could increase our on-time delivery.” You can focus on one end so there can be a payback from focusing on one narrow business outcome.
ebizQ: That makes sense. Now can you give me some real-world examples of how complex event processing and business activity monitoring can actually benefit a company?
Sure. The sweet spot of complex event processing is very, very low latency temporal pattern detection. “Temporal” just means “in time,” detecting patterns in time. I already mentioned the Wall Street example, algorithmic trading. [CEP] is used in all of those trading houses for algorithmic trading.
But the opposite of that is fraud detection. Regulatory authorities are using complex event processing to detect fraud. If this trade happens and this money moves to this bank account, hmm, we might have some fraud, because there's a sequence of events there that may indicate fraud. So you can imagine that same approach can be used on web traffic and for security purposes. So those are two obvious examples.
Some less obvious examples are in transportation and logistics, which is an extraordinarily event-driven business, or the airline industry, where if a certain event happens, it can cascade and cause problems. [For instance,] a missed flight can cause missed baggage.
There's a whole cascade of events that can happen in a business, in any business that is very event-heavy. Think of manufacturing, where the consequences of something going wrong can have a domino effect. Then CEP is a good fit.
ebizQ: What are some of the key best practices for companies adopting complex event processing?
They need to be able to find models. Let me explain that. Complex event processing is an execution platform. So you give it a pattern and you say “Find this pattern when it happens in large streams of data, high-velocity data. Given this pattern, I want you to find it.” That's what CEP does.
But what CEP doesn't do is it doesn't tell you what pattern you need to find. So you need analytic work. It might be some predictive analytics or advanced analytics, or even business rules to figure out what those patterns are that matter. That’s kind of the first step.
So many people think that CEP does real-time analytics. No, it's an execution engine. You can create a model that will help you create real-time dashboards and detect systems, but you have to find the model offline.
ebizQ: What would you say are some of the key best practices, then, for business activity monitoring?
The first one is to find all the business outcomes that you want to improve. It’s great to start with [key performance indicators (KPIs)] because that's the low= hanging fruit. So you make a list of all those key performance indicators.
Then you're going to have to go up and down, looking at the KPIs, looking at the data. [You need to] create a shorter list of initial KPIs because you don't want this big-bang project where you say “All right, we want business activity monitoring for all of these KPIs.” That may be the goal, but your implementation could be very, very complicated. It could take a long time; you could struggle to find the data. So you want to prioritize it and phase it in so you get the value from it as quickly as possible.
The second best practice, from a technical standpoint, is what I mentioned before: data integration. You have to find the connectors. There are products out there now that are able to deal with an enormous amount of volume that prior technologies weren't [able to do so]. You may have heard of companies like Splunk, or Sumo Logic, or Tibco LogLogic, which actually collect enormous amounts of information from IT systems and then compress, store, analyze and create dashboards from that.
ebizQ: This sounds like fairly a revolutionary technology, which means mistakes are crucial to avoid. So what are some of the biggest mistakes, and how do you avoid them in either of these technologies?
Well, a lot of the vendors--not just the ones I mentioned, but CEP vendors, too--they do tend to sort of over-sell and over-simplify. There’s that point I made that you have to find the model that you want to detect for CEP. You just can't just hook CEP and say, “I'm going to put all this network traffic in here; I'm just going to have it detect fraud.” No, you have to have an expert that knows what you're looking for.
So that's a key thing to understand, that's the key step back that you need to take before you adopt these technologies.
Other than that, you also have to look at how you are going to get this information to the people or other systems that need it. So fine, you've monitored something, you've detected something of interest. What are you going do about it? Can you do anything about it?
You might look for systems that can hook this up to existing application systems, expose it as a Web service, create dashboards for people to monitor, [or create] alerting systems so you have to look at the output of these systems and how you can actually connect it to the actions that you want to take in the business.
About the Author
Peter Schooff is a former contributing editor for ebizQ, where he also managed the ebizQ Forum for several years. Previously, Peter managed the database operations for a major cigar company, served as writer/editor of an early Internet entertainment site and developed a computer accounting system for several retail stores. Peter can be reached at firstname.lastname@example.org.More by Peter Schooff
ebizQ is the insider’s guide to next-generation business process management. We offer a growing collection of independent editorial articles on BPM trends, issues, challenges and solutions, all targeted to business and IT BPM professionals.
We cover BPM standards, governance, technology and continuous process improvement, as well as process discovery, modeling, simulation and optimization, among many other areas. We follow case management, decision management, business rules management, operational intelligence, complex event processing and other related topics. We closely track important trends such as the rise of social BPM, mobile BPM and BPM in the cloud. We also explore BPM’s use in functional areas, such as supply chain and customer management, and in key verticals, such as financial services, health care, insurance and government.
ebizQ's other BPM-oriented content includes podcasts, webcasts, webinars, white papers, a variety of expert blogs, a lively online forum and much more.