In the quest for massive data scalability, performance, and reliability, as a data management vendor, we anticipate an overall trend this year toward end-to-end system architectures based on data partitioning and dataflow parallelism, creating elastic data-data management architectures capable of adapting and scaling to existing and future business data requirements.

Based on hardware trajectories and customer appetite for larger and larger data processing, we predict the following trends for 2008:

Trend 1: Creation of TB scalable data management systems that are based entirely on main-memory storage.

Main-memory data management approaches are becoming mainstream in highly competitive business domains like the finance vertical, where low-latency, extreme transaction processing is essential for competitive advantage. Customers are inevitably looking to scale-out their current memory-based systems. There are numerous ways to increase scalability:

(1) Vertically scale-up using state-of-the-art many-core hardware coupled with ample memory
bandwidth
(2) Replicate data to another machine and load balance processing
(3) Partition data by dividing twice the work across two machines
(4) Compress data so more of it can fit into memory storage.

An example of (1) could be replacing a medium-powered server machine with a custom compute appliance like the Azul system for running Java applications. Hardware economics and application contention for shared resources measured through performance benchmarks will ultimately decide if this is a viable strategy for scalability and on-demand capacity management. This scalability approach, however, offers no secondary benefit of improved fault tolerance through distribution. Also, beefing up a node's memory and CPU resources does not necessarily increase network bandwidth which might be the scalability bottleneck.