Complex Event Processing
The Dawn of the Explosion: Complex Event Processing
By Don DeLoach, President and CEO, Aleri
Amid the widespread panic that was going around the financial markets over
the past couple of months, a banker in Paris said, "Banks are reacting
emotionally to this. The actual issue is in fact, quite small. However, it is
the emotional response that has banks reluctant to extend credit at this moment."
After a long pause, he added, "If this is what happens now, what will we
do when the crisis is real?" Wow. I hadn't seen that one coming. He was
actually asking just the right question. We will return to this later.
We live in a tabloid society. Globally. Dramatic stories, preferably about
violence, tragedy, sex, money, or ideally all of the aforementioned in combination
are what sells. So is it any wonder that vendors turn to hype when a new market
emerges? And while the market is emerging, we vendors compete for mindshare.
In order to do that, we have to get you to listen. As you get bombarded, you
will naturally filter down most of what comes your way. So our job is to convey,
as quickly and dramatically as possible, why you cannot live without our offerings.
And here lies the dilemma: if we convey a quick dramatic story, you may read
it, but are conditioned to not believe it, but if we convey a substantive discussion
about this market, it lacks the zest to qualify as interesting to most.
The trouble here is not unlike the trouble in the financial markets, which
is how to distinguish the story, the emotion, the hype from the real deal. And
most people are appropriately cynical when the message being conveyed prompts
action that benefits the messenger. Herein lies the conundrum that you face
though: just because the messenger benefits does not invalidate the message,
it just makes it suspect.
In that light, I have a thesis, which you should find suspect. My thesis is
that the Complex Event Processing (CEP) market is going to be huge. We are on
the dawn of this explosion, and I feel it is surely coming. Now, does my company
benefit if this comes true? Most certainly we will. So let me see if I can make
a fairly quick case for why I think this, and perhaps you may agree. Perhaps
not. At a minimum, you should be cynical.
I think the CEP market is going to be huge because it is being driven by a
few unmistakable elements and irreversible trends. The volume of data that organizations
need to deal with is growing and the time frames in which they need to act are
shrinking. CEP is specifically designed to absorb large amounts of data in real-time
and analyze that data on the fly. Ok, so now you are wondering how to get your
hands on CEP for your organization as soon as possible. Right?
For you skeptics out there, let's drill down a little more. If I were a skeptic,
I would be asking several questions. First, I would ask if this type of computing
can really do what it is purported to do, and I would require examples. Second,
I would want to see that the theoretical capabilities touted by the suppliers
could be delivered within the framework of the practical considerations needed
for any sort of widespread adoption. There are countless examples of technology
that had promise but never really amounted to the vast market it was projected
to address. I would also ask why this is any different from the traditional
technology. And what is it about the conditions in the market now that require
something else? I would ask how practical is it to introduce this new technology,
knowing that the switching costs often are prohibitive in and of themselves.
I would ask how practical it will be for the average organization to use this
technology. And I would be skeptical. My baseline is relational databases, ETL
and communication tools that we know work, provided by multi-billion dollar
companies that are proven and know what they are doing. Call me a cynic, I just
wouldn't buy it. If I were a skeptic, of course.
The thing is, I am a skeptic. And now that I am immersed in this technology
I have asked all of these questions. I almost opted out of this business a couple
of years ago, not because I didn't believe the technology was real, but because
I didn't believe it could be real in the "real world", where pragmatic
considerations make a "real" difference. But I stayed, because it
is real, and it can be pragmatic. We made it real, then our customers made it
really real. And most of all, the need for this technology is coming on like
a freight train and we are laying on the tracks.
It helps to look at precedents. When have other emerging technology markets
made it big? The closest illustration to CEP is relational databases and Client-Server
computing. In fairness, if we look at this in that context, we should be looking
at CEP and Event Driven Architectures. The two go hand in hand. So when you
look back to the late 1980's, early 1990's, you see the evolution of Client
Server and relational database technology. Oracle was an infant. The System-R
team, lead by Dr. Frank King from IBM, had recently completed their work pioneering
SQL, but the general market had not embraced the technology. And while Frank
went on to drive DB2 forward for IBM, the immediate future did not embrace IBM
DB2, or Oracle, or Sybase, or any other relational database for the mainstream.
Sybase pioneered stored procedures and triggers, really the technology heart
of Client-Server Computing, yet before it was mainstream, it was vendor hype.
Hype - that is what many thought. But in a time frame far shorter than any
of the wise, educated, tech-savvy people contemplated, relational databases
and client server became mainstream. Peoplesoft became real. Oracle became real.
Wall Street ran on Sybase. SAP moved from the Mainframes with R2 to the global
rage with R3. Client server and relational databases were more flexible and
nimble, able to adapt and grow with far greater ease, and ultimately with far
less cost that their predecessor technology. It made sense from a business,
cost and technology standpoint. And so the technology market moved, in a multi-billion
dollar way, to relational databases and client server.
Relational databases have their limitations. They are mainly around performance
and scalability. That is not to say that they are not per formant or that they
do not scale. They do - to a point. But the point traditional infrastructure
breaks down is in dealing with a ton of incoming data from internal and external
sources and when business needs require real time understanding of this information.
Today, there are use-cases growing by the day in numerous markets where the
amount of incoming data and the speed with which that data must be analyzed
and acted upon is growing well beyond what the existing technology can support.
And while all CEP solutions out there will not necessarily address the range
of the use cases, some will. The ones that won't will be challenged to expand
by improving their ability to handle changing data, by improving their storage
management capabilities, and by improving their ability to handle conditional
There are certainly illustrations where CEP solutions reflect both the elegance
of this new architecture and the pragmatic considerations of putting new technology
to work in a production class environment for the biggest organizations in the
world. Aleri, as you might expect me to argue, offers what I truly believe to
be the best example of this. And customers using the technology like Barclays,
HSBC, Dexia, Commerzbank, and many others stand in testimony to this fact.
Markets are bounding forward in a way where the underlying characteristics shout
out for CEP. Look at the increase in cell phone usage and voice over IP (VoIP)
and consider the growth in real-time telco applications. Consider the increase
in RFID and retail technology and consider the vast amount of increasing real
time data the industry is absorbing. Consider homeland security and the wealth
of well publicized, if not somewhat frightening imperative of the scrutiny of
security related information. And probably leading the way are financial services
in general, and the capital markets specifically.
If you look across the trade life cycle, there are a number of use-cases which
cry out for CEP. Market data absorption and enrichment, algorithmic trading,
smart order routing, market making and pricing engines, market liquidity discovery,
market surveillance, transaction cost analysis, regulatory and compliance requirements,
and especially real-time risk management. With the unmistakable trend to electronic
trading, penny option pricing, market fragmentation, and multi-asset and multi-geography
trading, there is the combination of growing data sources, volumes, and complexity
combined with the need to understand what is happening in real-time.
The other consideration is the architecture. I mentioned the need for event
driven architecture. This assumed that individual silos can contribute events
into a decoupled environment, where those events can be processed by other systems.
CEP lends itself beautifully to this architecture. Market data represents a
fast stream of external events, but internal silos reflect internal events which
can also be fed into and absorbed within the architecture. Using this approach,
CEP (or better stated, the right CEP) can stripe across internal silos as well
as absorb external market information and normalize, aggregate, and analyze
that information in real-time without having to disrupt the underlying silos.
Remember the banker we spoke of at the start. He said that the markets were
acting emotionally now, and that the actual problem was fairly small. But then
he went on to ask "but what will we do when it's real"? The obvious
answer would be that establishing CEP based systems for monitoring activities
would yield a more thorough understanding of an environment and thus give the
bank a time based edge over those that do not have such capabilities, so when
the (some think inevitable) real crisis hits, the bank using CEP will have an
advantage. Perhaps. It could also be said that in a world with two parties (at
least) on every transaction, even one in the dark can present a problem. However,
the real question, though, is what can be done to avert a crisis? This is when
CEP takes on a much more significant role. What everyone is coming around to
is the need for market transparency. That removes much of the risk in the market.
That is the great equalizer. CEP can provide greater transparency and visibility
of information in real time, across the enterprise, with out having to change
the underlying silos.
In the end, the question isn't so much if CEP will become an accepted technology
with a large market. It will. The question will really be, "can CEP grow
fast enough to avert a crisis that would otherwise be brought on by a lack of
transparency". For that matter, can it aid the financial, telco, retail,
and homeland security initiatives fast enough to enable them to avoid major
catastrophes in the market?
It can. But will it?
Hype? Maybe. But I hope not. I am hoping that the future of CEP is bright.
That is almost a certainty. My hope is that it is explosive; bright enough to
be entrenched before we wish we had acted sooner.
This is not clever technology, it is not useful technology, it is necessary
About the Author
Don DeLoach is the President and CEO of Aleri. Don brings over 20 years of experience within managing and growing software technology companies. Prior to joining Aleri, Don was President and CEO of YOUcentric, a leading provider of enterprise Customer Relationship Management software, until November 2001 when YOUcentric was acquired by J.D. Edwards. During his tenure, YOUcentric raised significant venture capital and substantially grew their revenue and customer base.
Prior to YOUcentric, Don spent five years at Sybase, a provider of database, middleware, and development tools, most recently as the Vice President for North American Geographic Sales, Telesales, Channel Sales, and Marketing. Additionally, Don held various management and sales positions with Prime Computer, Bull Information Systems, and Hitachi Data Systems.
Beyond his committed focus to providing leadership and driving company growth at Aleri, Don is also committed to supporting the business community. He has served on several corporate and industry boards including Apropos Technology, Broadbeam, and Illinois Information Technology Association (ITA). He is also a charter member of Tie Midwest and on the Board of the Juvenile Protection Association. Additionally, Don has contributed to various financial and technology trade publications and has provided content for several books including; The Handbook of Investment Technology and Investor Relations.
Don has also presented at numerous industry trades and conferences, including recent events including Gartnerís first annual Event Processing Summit, The World Financial Information Conference, and Financial Times Electric Money Conference.
Don holds a degree in Industrial and Systems Engineering from Georgia Tech.More by Don DeLoach
Aleri is the leading provider of enterprise-class complex event processing technology for financial institutions and beyond. Aleriís superior Streaming Platform is backed by the companyís deep background and knowledge gained over 20 years of supporting mission critical banking applications for the worldís largest banks and close to 10 years of pioneering research in the field of event processing.
The Aleri Platform was designed from the ground up to provide the most robust architecture available for the rapid implementation of mission critical applications within the most demanding environments. Built for high throughput with minimal latency, Aleriís event processing technology allows customers to analyze and respond instantly to high-volume, high-speed data to minimize risk and increase competitive advantage. Aleri is the first to develop and deploy a commercial enterprise-class application built on event processing technology, the Aleri Liquidity Management System, which is used by some of the largest global bank treasuries in the world.
Aleri is a global company headquartered in Chicago with offices in New Jersey, London, and Paris. For more information, visit www.aleri.com.