The Dawn of the Explosion: Complex Event Processing

Amid the widespread panic that was going around the financial markets over the past couple of months, a banker in Paris said, "Banks are reacting emotionally to this. The actual issue is in fact, quite small. However, it is the emotional response that has banks reluctant to extend credit at this moment." After a long pause, he added, "If this is what happens now, what will we do when the crisis is real?" Wow. I hadn't seen that one coming. He was actually asking just the right question. We will return to this later.

We live in a tabloid society. Globally. Dramatic stories, preferably about violence, tragedy, sex, money, or ideally all of the aforementioned in combination are what sells. So is it any wonder that vendors turn to hype when a new market emerges? And while the market is emerging, we vendors compete for mindshare. In order to do that, we have to get you to listen. As you get bombarded, you will naturally filter down most of what comes your way. So our job is to convey, as quickly and dramatically as possible, why you cannot live without our offerings. And here lies the dilemma: if we convey a quick dramatic story, you may read it, but are conditioned to not believe it, but if we convey a substantive discussion about this market, it lacks the zest to qualify as interesting to most.

The trouble here is not unlike the trouble in the financial markets, which is how to distinguish the story, the emotion, the hype from the real deal. And most people are appropriately cynical when the message being conveyed prompts action that benefits the messenger. Herein lies the conundrum that you face though: just because the messenger benefits does not invalidate the message, it just makes it suspect.

In that light, I have a thesis, which you should find suspect. My thesis is that the Complex Event Processing (CEP) market is going to be huge. We are on the dawn of this explosion, and I feel it is surely coming. Now, does my company benefit if this comes true? Most certainly we will. So let me see if I can make a fairly quick case for why I think this, and perhaps you may agree. Perhaps not. At a minimum, you should be cynical.

I think the CEP market is going to be huge because it is being driven by a few unmistakable elements and irreversible trends. The volume of data that organizations need to deal with is growing and the time frames in which they need to act are shrinking. CEP is specifically designed to absorb large amounts of data in real-time and analyze that data on the fly. Ok, so now you are wondering how to get your hands on CEP for your organization as soon as possible. Right?

For you skeptics out there, let's drill down a little more. If I were a skeptic, I would be asking several questions. First, I would ask if this type of computing can really do what it is purported to do, and I would require examples. Second, I would want to see that the theoretical capabilities touted by the suppliers could be delivered within the framework of the practical considerations needed for any sort of widespread adoption. There are countless examples of technology that had promise but never really amounted to the vast market it was projected to address. I would also ask why this is any different from the traditional technology. And what is it about the conditions in the market now that require something else? I would ask how practical is it to introduce this new technology, knowing that the switching costs often are prohibitive in and of themselves. I would ask how practical it will be for the average organization to use this technology. And I would be skeptical. My baseline is relational databases, ETL and communication tools that we know work, provided by multi-billion dollar companies that are proven and know what they are doing. Call me a cynic, I just wouldn't buy it. If I were a skeptic, of course.

The thing is, I am a skeptic. And now that I am immersed in this technology I have asked all of these questions. I almost opted out of this business a couple of years ago, not because I didn't believe the technology was real, but because I didn't believe it could be real in the "real world", where pragmatic considerations make a "real" difference. But I stayed, because it is real, and it can be pragmatic. We made it real, then our customers made it really real. And most of all, the need for this technology is coming on like a freight train and we are laying on the tracks.

It helps to look at precedents. When have other emerging technology markets made it big? The closest illustration to CEP is relational databases and Client-Server computing. In fairness, if we look at this in that context, we should be looking at CEP and Event Driven Architectures. The two go hand in hand. So when you look back to the late 1980's, early 1990's, you see the evolution of Client Server and relational database technology. Oracle was an infant. The System-R team, lead by Dr. Frank King from IBM, had recently completed their work pioneering SQL, but the general market had not embraced the technology. And while Frank went on to drive DB2 forward for IBM, the immediate future did not embrace IBM DB2, or Oracle, or Sybase, or any other relational database for the mainstream. Sybase pioneered stored procedures and triggers, really the technology heart of Client-Server Computing, yet before it was mainstream, it was vendor hype.

Hype - that is what many thought. But in a time frame far shorter than any of the wise, educated, tech-savvy people contemplated, relational databases and client server became mainstream. Peoplesoft became real. Oracle became real. Wall Street ran on Sybase. SAP moved from the Mainframes with R2 to the global rage with R3. Client server and relational databases were more flexible and nimble, able to adapt and grow with far greater ease, and ultimately with far less cost that their predecessor technology. It made sense from a business, cost and technology standpoint. And so the technology market moved, in a multi-billion dollar way, to relational databases and client server.

Relational databases have their limitations. They are mainly around performance and scalability. That is not to say that they are not per formant or that they do not scale. They do - to a point. But the point traditional infrastructure breaks down is in dealing with a ton of incoming data from internal and external sources and when business needs require real time understanding of this information. Today, there are use-cases growing by the day in numerous markets where the amount of incoming data and the speed with which that data must be analyzed and acted upon is growing well beyond what the existing technology can support. And while all CEP solutions out there will not necessarily address the range of the use cases, some will. The ones that won't will be challenged to expand by improving their ability to handle changing data, by improving their storage management capabilities, and by improving their ability to handle conditional situations.

There are certainly illustrations where CEP solutions reflect both the elegance of this new architecture and the pragmatic considerations of putting new technology to work in a production class environment for the biggest organizations in the world. Aleri, as you might expect me to argue, offers what I truly believe to be the best example of this. And customers using the technology like Barclays, HSBC, Dexia, Commerzbank, and many others stand in testimony to this fact.
Markets are bounding forward in a way where the underlying characteristics shout out for CEP. Look at the increase in cell phone usage and voice over IP (VoIP) and consider the growth in real-time telco applications. Consider the increase in RFID and retail technology and consider the vast amount of increasing real time data the industry is absorbing. Consider homeland security and the wealth of well publicized, if not somewhat frightening imperative of the scrutiny of security related information. And probably leading the way are financial services in general, and the capital markets specifically.

If you look across the trade life cycle, there are a number of use-cases which cry out for CEP. Market data absorption and enrichment, algorithmic trading, smart order routing, market making and pricing engines, market liquidity discovery, market surveillance, transaction cost analysis, regulatory and compliance requirements, and especially real-time risk management. With the unmistakable trend to electronic trading, penny option pricing, market fragmentation, and multi-asset and multi-geography trading, there is the combination of growing data sources, volumes, and complexity combined with the need to understand what is happening in real-time.

The other consideration is the architecture. I mentioned the need for event driven architecture. This assumed that individual silos can contribute events into a decoupled environment, where those events can be processed by other systems. CEP lends itself beautifully to this architecture. Market data represents a fast stream of external events, but internal silos reflect internal events which can also be fed into and absorbed within the architecture. Using this approach, CEP (or better stated, the right CEP) can stripe across internal silos as well as absorb external market information and normalize, aggregate, and analyze that information in real-time without having to disrupt the underlying silos.

Remember the banker we spoke of at the start. He said that the markets were acting emotionally now, and that the actual problem was fairly small. But then he went on to ask "but what will we do when it's real"? The obvious answer would be that establishing CEP based systems for monitoring activities would yield a more thorough understanding of an environment and thus give the bank a time based edge over those that do not have such capabilities, so when the (some think inevitable) real crisis hits, the bank using CEP will have an advantage. Perhaps. It could also be said that in a world with two parties (at least) on every transaction, even one in the dark can present a problem. However, the real question, though, is what can be done to avert a crisis? This is when CEP takes on a much more significant role. What everyone is coming around to is the need for market transparency. That removes much of the risk in the market. That is the great equalizer. CEP can provide greater transparency and visibility of information in real time, across the enterprise, with out having to change the underlying silos.

In the end, the question isn't so much if CEP will become an accepted technology with a large market. It will. The question will really be, "can CEP grow fast enough to avert a crisis that would otherwise be brought on by a lack of transparency". For that matter, can it aid the financial, telco, retail, and homeland security initiatives fast enough to enable them to avoid major catastrophes in the market?

It can. But will it?

Hype? Maybe. But I hope not. I am hoping that the future of CEP is bright. That is almost a certainty. My hope is that it is explosive; bright enough to be entrenched before we wish we had acted sooner.

This is not clever technology, it is not useful technology, it is necessary technology. Now.

About the Author

Don DeLoach is the President and CEO of Aleri. Don brings over 20 years of experience within managing and growing software technology companies. Prior to joining Aleri, Don was President and CEO of YOUcentric, a leading provider of enterprise Customer Relationship Management software, until November 2001 when YOUcentric was acquired by J.D. Edwards. During his tenure, YOUcentric raised significant venture capital and substantially grew their revenue and customer base. Prior to YOUcentric, Don spent five years at Sybase, a provider of database, middleware, and development tools, most recently as the Vice President for North American Geographic Sales, Telesales, Channel Sales, and Marketing. Additionally, Don held various management and sales positions with Prime Computer, Bull Information Systems, and Hitachi Data Systems. Beyond his committed focus to providing leadership and driving company growth at Aleri, Don is also committed to supporting the business community. He has served on several corporate and industry boards including Apropos Technology, Broadbeam, and Illinois Information Technology Association (ITA). He is also a charter member of Tie Midwest and on the Board of the Juvenile Protection Association. Additionally, Don has contributed to various financial and technology trade publications and has provided content for several books including; The Handbook of Investment Technology and Investor Relations. Don has also presented at numerous industry trades and conferences, including recent events including Gartnerís first annual Event Processing Summit, The World Financial Information Conference, and Financial Times Electric Money Conference. Don holds a degree in Industrial and Systems Engineering from Georgia Tech.

More by Don DeLoach

About Aleri

Aleri is the leading provider of enterprise-class complex event processing technology for financial institutions and beyond. Aleriís superior Streaming Platform is backed by the companyís deep background and knowledge gained over 20 years of supporting mission critical banking applications for the worldís largest banks and close to 10 years of pioneering research in the field of event processing. The Aleri Platform was designed from the ground up to provide the most robust architecture available for the rapid implementation of mission critical applications within the most demanding environments. Built for high throughput with minimal latency, Aleriís event processing technology allows customers to analyze and respond instantly to high-volume, high-speed data to minimize risk and increase competitive advantage. Aleri is the first to develop and deploy a commercial enterprise-class application built on event processing technology, the Aleri Liquidity Management System, which is used by some of the largest global bank treasuries in the world. Aleri is a global company headquartered in Chicago with offices in New Jersey, London, and Paris. For more information, visit