The Challenge of Sensing Events

Increasing numbers of probes and sensors mean that not only data volumes are on the rise, but the speed of data generation and the number of potential data sources in any focused business process are on the rise. The financial services industry has become the poster child of the convergence of these three trends. The typical financial services organization processes upward of 150,000 events per second, generated by an average of 10 different source systems. But events abound in other industries as well.

Today's manufacturing lines monitor production through complex networks of thousands of sensors evaluating processing times, product weights, and production temperatures. It would not be unusual to see a manufacturer generating upward of 35,000 events per second in this type of an environment.

Events are also proliferating in retail. In store locations, point-of-sale systems create tens of thousands of events per second. Throughout the broader retail supply chain, RFID technology puts retailers on a path to emitting hundreds of thousands of item reads per second. One recent retail industry study for RFID tagging at the item level estimated that 3 terabytes of events would be created in just one week at just one store location. The need to sense these events, filter them, and determine their relevance before in order to make an appropriate business response is a significant market driver for a new class of technology called complex event processing.

Event processing is not new; for 10 to 15 years, simple event processing has been common. Database triggers, thin-client user interfaces, and message-oriented middleware are all forms of event-based processing. But the output of these technologies is low-level system information and processing. These simple events are far removed from real business decisions like when to stop a potentially fraudulent banking transaction, when to pause production on the manufacturing floor, or when to pull a perishable retail product from the shelves because it can no longer safely be consumed.