Traditional relational databases and data warehouses provide Business Intelligence
(BI) that is of great value for conventional analytic applications where business
decisions need to be made today, this hour, or within minutes. However, these
traditional database resources cannot effectively support a new and growing class
of fast-paced business applications where decisions must be made this second,
or within milliseconds.
Instantaneous reaction to real-world event data can make a significant difference
in a variety of applications, including network intrusion detection and prevention,
airplane flight control, identity theft prevention, credit card fraud identification,
eBusiness transactions, and stock market trading.
For instance, in financial services more data is being generated faster than
ever before in stock exchanges, in part because of global market and regulatory
changes. Traders who can run quantitative analytics and automatically trade
from high-volume, high-speed data feeds gain critical competitive advantage.
In telecommunications and networking, as the number of customers served and
their usage of system resources explodes, maintaining system uptime through
split-second response and remediation to network intrusion or breakdown becomes
a critical necessity. And, in government and military applications, access to
critical complex analytics in real-time can make the difference between survival
or defeat on the battlefield, or can help avert domestic or international terrorist
The need to react and respond instantly to high-volume, high-speed data for
competitive advantage, to minimize risks, and avert disasters has never been
greater. Thus, a revolutionary new technology is needed to perform complex analytics
on high-speed, high-volume data to enable instantaneous response. That new technology
is high-performance Complex Event Processing (CEP).
High-performance Complex Event Processing
High-performance Complex Event Processing is the continuous processing, analysis,
and response to high-volume, high-speed data streams in real-time. It can be
used to identify and analyze cause-and-effect relationships among events, and
because it does all of this in real-time, it enables instant and informed responses
to critical intelligence in real-time.
Four different approaches are typically considered for high-performance Complex
Custom-Coded Applications are typically written in Java, C++ or other similar
programming languages. Until the recent advent of off-the-shelf CEP, this
had been the leading solution for many organizations. However, it is limited
by long development cycles (e.g. 6-9 months or more), high cost of maintenance,
Rules Engines watch input data streams for any conditions of interest that
match its rules - typically condition/action pairs, usually expressed using
"if-then" notation. The rules are typically written in a proprietary
language, and when a condition of a rule is matched, the rule is said to "fire"
a corresponding action, alert or output to external applications.
In-Memory Databases reduce the latency delays of traditional database management
systems by eliminating disk I/O, caching, and transaction logs, but they still
incur latency penalties and require custom-coding to build CEP applications.
Stream Processing Software is specifically designed to handle streaming
data, performing queries and computations on incoming messages as they fly
by, without storing them. Stream processing uses specialized primitives and
constructs, such as time-windows, to express stream-oriented processing logic.
Of all of the commercial offerings, high-performance CEP using stream processing
software offers the lowest latency, highest throughput, most flexible programming
model, and fastest time-to-value. Two types of high-performance CEP software
are generally available today: Platforms and Point Solutions.
High-performance CEP Platforms
In the same way that a commercial relational database includes core platform
capabilities which allow building a variety of server-based applications, high-performance
CEP platforms have a number of fundamental characteristics which enable developers
to build applications across a range of industries and problems.
To process information in real-time with lowest latency, a CEP platform must
be able to perform message processing without having to first store and retrieve
the data. The highest performance CEP platforms do this by using a language
such as StreamSQL, which extends standard SQL via time or event-based windows
to readily execute queries and perform pattern-matching functions on streaming
data. CEP platforms also must be capable of storing and accessing current or
historical state information, preferably using a familiar standard such as SQL.
In addition to these characteristics, a CEP platform should have the following
Predictable Outcomes: In a real-time processing infrastructure, the system
must process time-series records in a consistent, deterministic manner to
ensure that running the same dataset through the system multiple times always
yields the same results. Calculations performed on one time-series record
should not interfere with the calculations performed on another. And, all
computations should perform in the correct sequence, even when spread across
Data Safety and Availability: If a failure occurs in a CEP application -
regardless of the cause (e.g. hardware, operating system, software, application)
- the application needs to continue operating without losing any data. This
is especially true in real-time environments where the value of the information
may only exist for fractions of a second.
Automatic Scaling and Partitioning: As the volume of real-time information
continues to expand, applications for processing this data must be able to
scale quickly and automatically. These applications must also be free of artificial
limitations created by system architecture. To do this successfully, an application
must be easily split over multiple processors or clusters without user intervention.
Flexible, Graphical Programming: Since much of the underlying CEP technology
is complicated in nature, stream processing development should be available
to business users and IT professionals at a high level of abstraction via
a drag-and-drop graphical workflow development paradigm, without requiring
low-level programming or deep specialized knowledge of the physical systems
or underlying infrastructure.
Security: As CEP becomes more broadly integrated throughout the enterprise,
users can reasonably expect these systems will provide appropriate security
capabilities and integrate with existing systems. At minimum, systems should
include secure network access to server components and integration with industry
leading LDAP servers for centralized management of user information, authorization,
Interoperability with Enterprise Systems and Data: The system must be able
to process a range of data types and formats including structured and unstructured
data, financial market data, IP logs, email, XML data and files, RSS feeds,
and comma-separated value (CSV) files. Out-of-the-box adapters should be available
to common data sources and messaging infrastructures, and the system should
include standard Java, C++, and .NET APIs which allow interfacing with other
systems or client applications.
A platform with these capabilities provides the necessary framework for building
a variety of applications within and across multiple industries.
High Performance CEP Point Solutions
On a smaller scale, CEP point solutions are designed to provide Complex Event
Processing for specific limited application areas, such as only one form of
algorithmic trading, or only e-Fraud. In general, these solutions have a proprietary
programming language and fixed or pre-defined data models, user interfaces,
Typically, businesses acquire CEP point solutions to solve a specific problem
limited to a departmental "silo," while Chief Information Officers
and Chief Technology Officers who see the potential for CEP to benefit multiple
departments or applications within their organizations will look to a CEP platform.
High-performance CEP Platforms offer flexibility, broader adaptability to various
business challenges, and cross-enterprise interoperability. Many applications
can be supported by one high-performance CEP platform -- typically at lower
cost per application than if individual point solutions were employed.
Real-World CEP: Transforming Business and Government BI Applications
High-performance CEP is transforming how organizations in financial services,
telecommunications, online gaming, and government agencies, acquire, process,
and manage real-time and historical event data.
Financial services firms leverage high-performance CEP to monitor data coming
into their ticker plants and feed handlers, trigger alerts if latency drops
below acceptable levels, normalize symbology, filter massive data streams to
pass only selected data downstream, route the first arriving quote from across
multiple exchanges to traders, and calculate real-time statistics on quality
of service from feeds. Additional applications include algorithmic trading,
regulatory compliance, real-time P&L, and enterprise risk management.
In cellular-based telephony and e-Business, fraud detection applications must
operate in real-time. By leveraging high-performance CEP's real-time processing
and analytic capabilities, service providers and e-Businesses can highlight
and halt suspicious traffic that may be fraudulent and could lead to damaging
Online gaming companies use high-performance CEP to monitor their game environments
in real-time, track player activity at rates of more than 30,000 messages/second,
immediately offer fixes for any game oddities, and to identify and react to
malicious player behavior in real-time to ensure game integrity.
Government agencies use high-performance CEP for data-gathering systems used
for intelligence, security, and defense that must be able to "sense and
respond" to patterns of events that indicate pending threats. High-performance
CEP forms the basis for communication or transaction monitoring systems that
protect national security by analyzing activity for terror suspect patterns
as they occur.
High-performance CEP: Powering 21st Century Business
The need to leverage intelligence from the massive amount of data hitting organizations
today is no longer a nice-to-have, but is rapidly becoming essential to maintaining
competitive advantage. It's time to get serious about the data deluge, and to
transform this data into intelligence. High-performance CEP is the solution
- and it's the only technology that can deliver on that promise.
About the Author
Bill Hobbib is Vice President, Marketing, at StreamBase Systems, Inc., the leader in the fast-growing Complex Event Processing (CEP) market. Hobbib is a former Executive Director of Industry Solutions Marketing, Product Marketing, and Field Marketing at Ascential Software. Previously, Hobbib was Vice President of Business Development at parallel-processing software provider, Torrent Systems, Prior to that Bill held management positions at Kenan Systems, Lucent Technologies, Siemens, Cognex Corp, and Open Market. He holds a B.S. degree from M.I.T. and an MBA from the University of Chicago Graduate School of Business.