Evolution of the Enterprise Service Bus (Part II of II)

*Editor's Note: For Part I of this article, click here.

So, what else is there?

Now that ESB has become an established product category, numerous ESB vendors are competing on architecture, connectivity options, ease of use and Quality of Service issues such as continuous availability. Since organizations have been putting ESBs to work in the real world for a few years now, there is better clarity on what other infrastructure is necessary to build, deploy, manage, and extract business value from a widespread SOA across an organization.

The list is fairly substantial, but Iíll focus on a few of the important ones Ė

  • †††††† Web Services Management and SOA governance
  • †††††††† Advanced Web Services (WS-*)
  • †††††††† Complex Event Processing
  • †††††††† Semantic Data Integration
SOA Management and Governance

The path toward SOA varies drastically at each company and sometimes even within different groups within the same company. An organization may have different development teams experimenting with Web services toolkits or using the Web services capabilities that come as part of their enterprise resource planning (ERP) system or their favorite application server environment. There may be multiple ESBs deployed or none at all.

A SOA Management platform provides visibility, security, control and policy enforcement across an end-to-end business process as the process executes across these many disparate environments. In addition to tracking Service Level Agreements (SLA), it may be able to provide business-level views in order to understand the business level impact of failures in a SOA. It may also be able to dynamically tune the SOA environment in order to enforce the SLA or enforce any other business policy. This is true whether the interaction styles are point to point or asynchronous and event driven.†

Depending on the implementation of the WSM platform, it may extend its reach across an ESB, application server platforms and database access; and it may even provide business level metrics and reporting at a business process level as the service execution traverses across different platforms, databases, protocols, and appliances (Figure 1).

Figure 1: SOA Management provides business process visibility across diverse infrastructures

Advanced Web Services (WS-*)

Advanced Web Services specifications such as WS-ReliableMessaging, WS-Security, WS-Addressing and WS-Policy are becoming a reality as they gain momentum and support from the community of vendors who participate in creating these specifcations and work to ensure interoperability with each other. In the past, ESBs have been labeled as proprietary, which has mostly been due to having a vendor-specific, message-oriented middleware layer as part of their core architecture. As ESBs are beginning to implement these advanced Web services protocols, the proprietary stigma is becoming a non-issue. In fact, an ESB can help make these WS-* specifications more real by providing an implementation on top of the ESBís architecture adding scalability, manageability and availability to advanced Web services. In addition, these specifications will make ESBs more capable of interoperating as other platforms and application vendors implement the specs.

Complex Event Processing

Complex Event Processing (CEP), sometimes known as Event Stream Processing (ESP), is a relatively new field in the area of Event Driven Architectures (EDA). Significant traction is already occurring for CEP in areas of algorithmic trading, fraud detection in financial services and supply chain management with Radio Frequency Identification (RFID) processing and filtering. CEP is about capturing and analyzing high volume streams of events, identifying sophisticated patterns and applying time aware event correlation and decision logic against those patterns. Typically, a CEP engine that is plugged into the ESB as a service will perform these tasks. The stream of events may come through the ESB or may be another source such as an external stock ticker feed or an RFID reader. The course of action taken when a complex pattern of events is identified may vary, but can range from alerting a business user in a Business Activity Monitoring (BAM) dashboard or to invoke a service or a business process via the ESB (Figure 2).

Figure 2: Realtime Monitoring, Dashboarding, and Alerting with CEP

Semantic Data Integration

An ESB makes it easy to translate disparate application data from one format to another by providing a means for inserting data transformation engines as services on an as-needed basis into business processes. Service patterns such as Validate, Enrich, Transform and Operate (VETO) have begun to emerge as a best practice for converting to and from a common data format to individual applicationís proprietary data formats.

In a process-oriented, asynchronous event driven environment, data transformations can not be dealt with in a point-to-point fashion. In order for an application to truly be decoupled from all others, the data being communicated needs to be translated from the applicationís proprietary format into a common or canonical data format that is used across the organization. In this fashion, each integration point of an application plugging into the SOA needs to only be concerned with how the data gets converted to and from the proprietary format to the canonical format. The transformation services associated with the other applications through the bus that take care of converting to the target data formats on an as-needed basis. This dramatically reduces the complexity of adding new applications into the SOA or changing the interaction patterns of the existing ones. The additional benefit of taking this approach is that mediation services such as routers, splitters, aggregators, etc. can be written to conform to the canonical format.

Mapping tools can create data translations to convert from one data format to another, but at the bigger picture whatís been typically missing is the means for managing the mappings and the relationships between large numbers of complex data models across a diverse set of applications and data sources. Semantic data mapping tools have begun to emerge as complementary technologies to ESB. This is the next big frontier that needs to be addressed by those building large scale SOAs. In a world of point-to-point integration, itís easy to imagine a set of point-to-point data transformations between each pair of applications that are communicating in a one-off fashion.†

Bringing it all together using an ESB

As we have learned through adoption and deployment of ESB technology by many enterprises, there is a lot to consider in addition to an ESB when planning your strategy for standardizing on SOA infrastructure. You will have lots of services hosted and a variety of interactions between platforms that need to be secured and governed.

Event-driven architectures are beginning to emerge as the best approach toward implementing a SOA where applications are truly decoupled from each other. Large scale SOA projects using an ESB are broadening the reach of application integration in ways that were never before possible. However, the challenge of building and managing the complex semantic mappings between disparate representations of enterprise data is daunting and is an area that will require continued research and new innovations.

In keeping with the true spirit of Service-Oriented Architecture, these technologies can all generally be used both standalone and pluggable into anything using Web services interfaces and can also be brought together in a common integration through an ESB.