High-Performance CEP: Driving Next-Generation Business Intelligence

Traditional relational databases and data warehouses provide Business Intelligence (BI) that is of great value for conventional analytic applications where business decisions need to be made today, this hour, or within minutes. However, these traditional database resources cannot effectively support a new and growing class of fast-paced business applications where decisions must be made this second, or within milliseconds.

Instantaneous reaction to real-world event data can make a significant difference in a variety of applications, including network intrusion detection and prevention, airplane flight control, identity theft prevention, credit card fraud identification, eBusiness transactions, and stock market trading.

For instance, in financial services more data is being generated faster than ever before in stock exchanges, in part because of global market and regulatory changes. Traders who can run quantitative analytics and automatically trade from high-volume, high-speed data feeds gain critical competitive advantage. In telecommunications and networking, as the number of customers served and their usage of system resources explodes, maintaining system uptime through split-second response and remediation to network intrusion or breakdown becomes a critical necessity. And, in government and military applications, access to critical complex analytics in real-time can make the difference between survival or defeat on the battlefield, or can help avert domestic or international terrorist attacks.

The need to react and respond instantly to high-volume, high-speed data for competitive advantage, to minimize risks, and avert disasters has never been greater. Thus, a revolutionary new technology is needed to perform complex analytics on high-speed, high-volume data to enable instantaneous response. That new technology is high-performance Complex Event Processing (CEP).

High-performance Complex Event Processing

High-performance Complex Event Processing is the continuous processing, analysis, and response to high-volume, high-speed data streams in real-time. It can be used to identify and analyze cause-and-effect relationships among events, and because it does all of this in real-time, it enables instant and informed responses to critical intelligence in real-time.

Four different approaches are typically considered for high-performance Complex Event Processing:

  • Custom-Coded Applications are typically written in Java, C++ or other similar programming languages. Until the recent advent of off-the-shelf CEP, this had been the leading solution for many organizations. However, it is limited by long development cycles (e.g. 6-9 months or more), high cost of maintenance, and inflexibility.
  • Rules Engines watch input data streams for any conditions of interest that match its rules - typically condition/action pairs, usually expressed using "if-then" notation. The rules are typically written in a proprietary language, and when a condition of a rule is matched, the rule is said to "fire" a corresponding action, alert or output to external applications.
  • In-Memory Databases reduce the latency delays of traditional database management systems by eliminating disk I/O, caching, and transaction logs, but they still incur latency penalties and require custom-coding to build CEP applications.
  • Stream Processing Software is specifically designed to handle streaming data, performing queries and computations on incoming messages as they fly by, without storing them. Stream processing uses specialized primitives and constructs, such as time-windows, to express stream-oriented processing logic.

Of all of the commercial offerings, high-performance CEP using stream processing software offers the lowest latency, highest throughput, most flexible programming model, and fastest time-to-value. Two types of high-performance CEP software are generally available today: Platforms and Point Solutions.

High-performance CEP Platforms

In the same way that a commercial relational database includes core platform capabilities which allow building a variety of server-based applications, high-performance CEP platforms have a number of fundamental characteristics which enable developers to build applications across a range of industries and problems.

To process information in real-time with lowest latency, a CEP platform must be able to perform message processing without having to first store and retrieve the data. The highest performance CEP platforms do this by using a language such as StreamSQL, which extends standard SQL via time or event-based windows to readily execute queries and perform pattern-matching functions on streaming data. CEP platforms also must be capable of storing and accessing current or historical state information, preferably using a familiar standard such as SQL.

In addition to these characteristics, a CEP platform should have the following capabilities:

  • Predictable Outcomes: In a real-time processing infrastructure, the system must process time-series records in a consistent, deterministic manner to ensure that running the same dataset through the system multiple times always yields the same results. Calculations performed on one time-series record should not interfere with the calculations performed on another. And, all computations should perform in the correct sequence, even when spread across parallel hardware.
  • Data Safety and Availability: If a failure occurs in a CEP application - regardless of the cause (e.g. hardware, operating system, software, application) - the application needs to continue operating without losing any data. This is especially true in real-time environments where the value of the information may only exist for fractions of a second.
  • Automatic Scaling and Partitioning: As the volume of real-time information continues to expand, applications for processing this data must be able to scale quickly and automatically. These applications must also be free of artificial limitations created by system architecture. To do this successfully, an application must be easily split over multiple processors or clusters without user intervention.
  • Flexible, Graphical Programming: Since much of the underlying CEP technology is complicated in nature, stream processing development should be available to business users and IT professionals at a high level of abstraction via a drag-and-drop graphical workflow development paradigm, without requiring low-level programming or deep specialized knowledge of the physical systems or underlying infrastructure.
  • Security: As CEP becomes more broadly integrated throughout the enterprise, users can reasonably expect these systems will provide appropriate security capabilities and integrate with existing systems. At minimum, systems should include secure network access to server components and integration with industry leading LDAP servers for centralized management of user information, authorization, and authentication.
  • Interoperability with Enterprise Systems and Data: The system must be able to process a range of data types and formats including structured and unstructured data, financial market data, IP logs, email, XML data and files, RSS feeds, and comma-separated value (CSV) files. Out-of-the-box adapters should be available to common data sources and messaging infrastructures, and the system should include standard Java, C++, and .NET APIs which allow interfacing with other systems or client applications.

A platform with these capabilities provides the necessary framework for building a variety of applications within and across multiple industries.

High Performance CEP Point Solutions

On a smaller scale, CEP point solutions are designed to provide Complex Event Processing for specific limited application areas, such as only one form of algorithmic trading, or only e-Fraud. In general, these solutions have a proprietary programming language and fixed or pre-defined data models, user interfaces, and analytics.

Typically, businesses acquire CEP point solutions to solve a specific problem limited to a departmental "silo," while Chief Information Officers and Chief Technology Officers who see the potential for CEP to benefit multiple departments or applications within their organizations will look to a CEP platform. High-performance CEP Platforms offer flexibility, broader adaptability to various business challenges, and cross-enterprise interoperability. Many applications can be supported by one high-performance CEP platform -- typically at lower cost per application than if individual point solutions were employed.

Real-World CEP: Transforming Business and Government BI Applications

High-performance CEP is transforming how organizations in financial services, telecommunications, online gaming, and government agencies, acquire, process, and manage real-time and historical event data.

Financial services firms leverage high-performance CEP to monitor data coming into their ticker plants and feed handlers, trigger alerts if latency drops below acceptable levels, normalize symbology, filter massive data streams to pass only selected data downstream, route the first arriving quote from across multiple exchanges to traders, and calculate real-time statistics on quality of service from feeds. Additional applications include algorithmic trading, regulatory compliance, real-time P&L, and enterprise risk management.

In cellular-based telephony and e-Business, fraud detection applications must operate in real-time. By leveraging high-performance CEP's real-time processing and analytic capabilities, service providers and e-Businesses can highlight and halt suspicious traffic that may be fraudulent and could lead to damaging financial losses.

Online gaming companies use high-performance CEP to monitor their game environments in real-time, track player activity at rates of more than 30,000 messages/second, immediately offer fixes for any game oddities, and to identify and react to malicious player behavior in real-time to ensure game integrity.

Government agencies use high-performance CEP for data-gathering systems used for intelligence, security, and defense that must be able to "sense and respond" to patterns of events that indicate pending threats. High-performance CEP forms the basis for communication or transaction monitoring systems that protect national security by analyzing activity for terror suspect patterns as they occur.

High-performance CEP: Powering 21st Century Business

The need to leverage intelligence from the massive amount of data hitting organizations today is no longer a nice-to-have, but is rapidly becoming essential to maintaining competitive advantage. It's time to get serious about the data deluge, and to transform this data into intelligence. High-performance CEP is the solution - and it's the only technology that can deliver on that promise.

About the Author

Bill Hobbib is Vice President, Marketing, at StreamBase Systems, Inc., the leader in the fast-growing Complex Event Processing (CEP) market. Hobbib is a former Executive Director of Industry Solutions Marketing, Product Marketing, and Field Marketing at Ascential Software. Previously, Hobbib was Vice President of Business Development at parallel-processing software provider, Torrent Systems, Prior to that Bill held management positions at Kenan Systems, Lucent Technologies, Siemens, Cognex Corp, and Open Market. He holds a B.S. degree from M.I.T. and an MBA from the University of Chicago Graduate School of Business.

More by Bill Hobbib