Business Activity Monitoring (BAM)
Going to Extremes: Extreme Transaction Processing (Part I of II)
Shivaji Sarkar, TATA Consultancy Services
Peter Mendis, TATA Consultancy Services
Extreme Transaction Processing - What & Why
Extreme Transaction Processing (XTP) is an exceptionally demanding form of transaction
processing. Transactions of most high-end (more than 10,000 concurrent accesses
or 500 transactions per second) or ultra-high-end (more than 100,000 concurrent
accesses or 5,000 transactions per second) requirements or more would require
this form of processing.
Gartner defines XTP as an application style aimed at supporting the design,
development, deployment, management and maintenance of distributed TP applications
characterized by exceptionally demanding performance, scalability, availability,
security, manageability and dependability requirements.
Very much like traditional TP systems, XTP applications are aimed at enabling
efficient, reliable concurrent and real-time access (read/update) to a shared
database by executing application programs commonly referred to as "transactions."
The first and foremost adoption of XTP can be observed amongst the financial
institutions (a one-millisecond advantage in trading applications can be worth
$100 million a year to a major brokerage firm, by one estimate), whose prime
requirements are more processing capability, but without requiring exponential
increase in hardware costs, in areas such as fraud detection, risk computation,
and stock trade resolution, to profit from minute, fleeting price anomalies
and to mask their intentions via "time-slicing," or carving huge orders
into smaller batches so as not to move the market.
What is Federated ESB (Enterprise Service Bus)?
Federated ESB (FESB) provides the ability to use multiple service registries
and administration domains while mapping the disparate registries into a federated
set of services. The federated ESB facilitates service interactions with multiple
ESB implementations. One master ESB to which several dependent ESBs are federated
to provide a subset of services that are applicable throughout the enterprise.
Fig 1. Federated Enterprise Service Bus
What is Grid Architecture?
Grid architecture aims to integrate, virtualize, and manage resources and services
within distributed, heterogeneous, dynamic "virtual organizations."
The realization of this goal requires the disintegration of computing systems
within and across organizations for resources to be accessed as and when required,
regardless of physical location.
The two key players in grid environment are resource providers and resource
consumers. Resource consumers adopt the strategy of solving their problems at
low cost within a required timeframe and resource providers adopt the strategy
of obtaining best possible return on their investment.
The supply-demand between resource providers and resource consumers are guided
by the following key functional and non-functional requirements:
i. Interoperability and support for dynamic and heterogeneous environments
ii. Optimization, ease of use and extensibility
iii. Quality of service (QOS) assurance
v. Scalability & availability
The utility provided by such grid infrastructure is realized as a set of capabilities:
request management & optimization, resource management & optimization,
and infrastructure optimization & federation.
Fig 2. Supply-Demand (click on image to enlarge)
Challenges in Adoption of XTP
The fundamental challenge of XTP is supporting business-critical requirements
comparable (or even more demanding) to those of classic TP systems, but in a
more distributed, interoperable, standards-based and global-class environment.
XTP requirements are fostering a new cycle of potentially disruptive innovation
in application platforms involving:
Atomicity - The effect of the execution of a transaction cannot be partial.
A transaction must be fully executed or its partial effects must be rolled back
in case of failure.
Consistency - Execution of a transaction cannot break the integrity rules
associated with the database.
Isolation - While a transaction is being executed, the data it is working on
is not visible by other applications, so no other application can have access
to data in an intermediate state
Durability - Once a transaction has been completed (or "committed"),
the corresponding database state change is persisted and cannot be rolled back.
Availability - Minimal planned system downtime for application or system
Reliability - Fast and transparent recovery from hardware/software failures
Manageability - Installation, configuration, administration, monitoring,
maintenance and change management
Security - Authentication, authorization, access control and communication
Performance - Execution of transactions within a pre-defined time frame
(typically subseconds), irrespective of the number of users concurrently accessing
Scalability - Support to large and fluctuating workloads by optimally
and dynamically allocating resources accordingly
Accountability - Tracking and tracing operations for auditing or other
Extensibility - Adding new transaction types rapidly without stopping
Faster Infrastructure - Dedicated compute resources, garbage collection
pauses, memory heap size, lock contentions limit scalable performance
Costs - Utilization, footprint, power and cooling
Leveraging Technology for XTP
From traditional TPMs and database management systems (DBMSs) to ESB technology
of today, technology has evolved dramatically to support high scalability and
performance, optimized resource allocation, fast recovery and transaction integrity
on top of the bare operating system and file system. However, the combination
of EAS and ESB technology cannot provide the scalability and performance required
by the most extreme scenarios without forcing substantial application redesign.
Fig 3. Technology Options for XTP (click on image to enlarge)
Fig 4. Incremental Technologies (click on image to enlarge)
Fig 5. Emerging Technologies (click on image to enlarge)
Check back Thursday next week for Part II to read about
Service Virtualization through Federated ESB.