Going to Extremes: Extreme Transaction Processing (Part I of II)

Untitled Document


Extreme Transaction Processing - What & Why

Extreme Transaction Processing (XTP) is an exceptionally demanding form of transaction processing. Transactions of most high-end (more than 10,000 concurrent accesses or 500 transactions per second) or ultra-high-end (more than 100,000 concurrent accesses or 5,000 transactions per second) requirements or more would require this form of processing.

Gartner defines XTP as an application style aimed at supporting the design, development, deployment, management and maintenance of distributed TP applications characterized by exceptionally demanding performance, scalability, availability, security, manageability and dependability requirements.

Very much like traditional TP systems, XTP applications are aimed at enabling efficient, reliable concurrent and real-time access (read/update) to a shared database by executing application programs commonly referred to as "transactions."

The first and foremost adoption of XTP can be observed amongst the financial institutions (a one-millisecond advantage in trading applications can be worth $100 million a year to a major brokerage firm, by one estimate), whose prime requirements are more processing capability, but without requiring exponential increase in hardware costs, in areas such as fraud detection, risk computation, and stock trade resolution, to profit from minute, fleeting price anomalies and to mask their intentions via "time-slicing," or carving huge orders into smaller batches so as not to move the market.

What is Federated ESB (Enterprise Service Bus)?

Federated ESB (FESB) provides the ability to use multiple service registries and administration domains while mapping the disparate registries into a federated set of services. The federated ESB facilitates service interactions with multiple ESB implementations. One master ESB to which several dependent ESBs are federated to provide a subset of services that are applicable throughout the enterprise.

 Extreme ESB Requestor

Fig 1. Federated Enterprise Service Bus

What is Grid Architecture?

Grid architecture aims to integrate, virtualize, and manage resources and services within distributed, heterogeneous, dynamic "virtual organizations." The realization of this goal requires the disintegration of computing systems within and across organizations for resources to be accessed as and when required, regardless of physical location.

The two key players in grid environment are resource providers and resource consumers. Resource consumers adopt the strategy of solving their problems at low cost within a required timeframe and resource providers adopt the strategy of obtaining best possible return on their investment.

The supply-demand between resource providers and resource consumers are guided by the following key functional and non-functional requirements:

i. Interoperability and support for dynamic and heterogeneous environments
ii. Optimization, ease of use and extensibility
iii. Quality of service (QOS) assurance
iv. Security
v. Scalability & availability

The utility provided by such grid infrastructure is realized as a set of capabilities: request management & optimization, resource management & optimization, and infrastructure optimization & federation.

Extreme ESB GiveItAlittlename

Fig 2. Supply-Demand (click on image to enlarge)

Challenges in Adoption of XTP

The fundamental challenge of XTP is supporting business-critical requirements comparable (or even more demanding) to those of classic TP systems, but in a more distributed, interoperable, standards-based and global-class environment. XTP requirements are fostering a new cycle of potentially disruptive innovation in application platforms involving:

Atomicity - The effect of the execution of a transaction cannot be partial. A transaction must be fully executed or its partial effects must be rolled back in case of failure.
Consistency - Execution of a transaction cannot break the integrity rules associated with the database.
Isolation - While a transaction is being executed, the data it is working on is not visible by other applications, so no other application can have access to data in an intermediate state
Durability - Once a transaction has been completed (or "committed"), the corresponding database state change is persisted and cannot be rolled back.
Availability - Minimal planned system downtime for application or system upgrades
Reliability - Fast and transparent recovery from hardware/software failures
Manageability - Installation, configuration, administration, monitoring, maintenance and change management
Security - Authentication, authorization, access control and communication encryption
Performance - Execution of transactions within a pre-defined time frame (typically subseconds), irrespective of the number of users concurrently accessing the system
Scalability - Support to large and fluctuating workloads by optimally and dynamically allocating resources accordingly
Accountability - Tracking and tracing operations for auditing or other management reasons
Extensibility - Adding new transaction types rapidly without stopping operations
Faster Infrastructure - Dedicated compute resources, garbage collection pauses, memory heap size, lock contentions limit scalable performance
Costs - Utilization, footprint, power and cooling

Leveraging Technology for XTP

From traditional TPMs and database management systems (DBMSs) to ESB technology of today, technology has evolved dramatically to support high scalability and performance, optimized resource allocation, fast recovery and transaction integrity on top of the bare operating system and file system. However, the combination of EAS and ESB technology cannot provide the scalability and performance required by the most extreme scenarios without forcing substantial application redesign.

Extreme ESB Technology

Fig 3. Technology Options for XTP (click on image to enlarge)

Incremental Technologies

Extreme ESB More Technology

Fig 4. Incremental Technologies (click on image to enlarge)

Emerging Technologies

Extreme ESB Technology Even More

Fig 5. Emerging Technologies (click on image to enlarge)

Check back Thursday next week for Part II to read about Service Virtualization through Federated ESB.