Evolution of the Enteprise Service Bus (Part I of II)

Over the past four years we have seen a great deal of favorable adoption of the Enterprise Service Bus (ESB) as a product category in the IT industry. The ESB has taken off and evolved in the ways it is implemented and deployed since the term was first coined in 2002. Numerous Application Platform Suite (APS) vendors such as IBM and BEA, Enterprise Application Integration (EAI) vendors such as TIBCO and webMethods and even Web Services toolkit vendors have all adopted the ESB moniker. Even British Telecom has embedded an ESB in a hardware box with their “BT Integrate” offering. We have also seen some significant changes in the way technologists in IT departments view the role of an ESB as an important part of their IT landscape. As is the case with most new technology categories, it has taken some time for the hype to subside. Now that the dust has settled, we can see the practical uses of ESBs as they get deployed in the real world.

Analyst views from firms such as Gartner and Forrester have also shifted from hailing ESB as the all-you’ll-ever-need for SOA to more of an implied part of the infrastructure in support of a SOA. Reports from these analysts and from thought-leading vendors have provided greater clarity on what makes up the definition of an ESB. The constant that remains is that an ESB is used to connect, mediate and control the interactions between a diverse set of applications that are exposed through the bus using service-level interfaces.

In my travels around the world talking with people about SOA and ESB, I have noticed a shift in people’s thinking and witnessed a trend of enlightenment over the past two years. The questions on most people’s mind has shifted from  “what is SOA?” and “What is an ESB and why would I need one?” to “I know I need a SOA, I know I need an ESB, but what is an ESB best suited for and what other technology infrastructure do I need in order to implement my SOA?”

Process-Oriented, Event Driven Architectures

Point-to-point integrations have typically been addressed with simple request-reply, synchronous style interactions. In this type of environment, a Web service intermediary acting as a proxy for data transformation and routing can work well. However, the real sweet spot where an ESB has shown its power and flexibility is in process-oriented, event driven architectures. When doing broad scale integrations across many disparate applications, the key to success is to have an architecture that allows for each application to be decoupled from the rest of the SOA by using the ESB as a form of mediation. Each application should be made to operate independently from the other by being capable of receiving individual tasks or units of work that are processed as asynchronous events. If there is another action to be taken as a result of receiving and processing the asynchronous event, then so be it. That action does not necessarily represent a response to a request that the original initiator of the action is waiting for. It’s more likely that the next action to occur is itself an asynchronous event that is placed back onto the bus to be forwarded along to the next application. The surrounding process manager that is part of the ESB, which is controlling the interactions between the individual invocations, is what is concerned with getting that event sent along to the next application.

The same is also true even if an application makes a request and expects a response in order to continue its processing. The request can be placed on the bus and sent across a distributed path of many applications, data sources, routers and transformers in accordance with the process that is defined for it. That series of actions may all be processed as independent events and the “response” may come later as an independent event.

The event-driven interaction style is a major advantage of keeping applications decoupled form one another. When plugged into an ESB, each application does not need to understand the intimate details of how to interact with all the other applications. The ESB handles all the protocols, data formats and different interaction styles.

Event-driven architectures only work effectively if certain conditions are in place. First, the ESB must have reliable asynchronous messaging and high availability capabilities. In a synchronous point-to-point integration scenario, if an application fails to receive a response from a request, it can code in some error handing and retry. In the asynchronous scenario, the application simply fires off an event to the ESB and forgets about it until something else occurs to trigger further processing. There will be many points where the whole business transaction will be nowhere else but in transit through the bus. The ESB must therefore be capable of surviving failure and also sustaining the essence of the business transaction’s complex topologies that may involve failures and recoveries along the way.

Another condition that has to be met is the applications themselves need to be written or adapted to this event driven style of interaction. In the case where the ordering of things is important, then each application may need to check or compensate for things occurring out of order or the ESB itself may need to be capable of ensuring order of events across complex deployment topologies and across failures and recoveries.