My post before Christmas sparked an excellent thread on the SOA Yahoo! Group around whether an Enterprise Service Bus was really a “keystone product in a SOA strategy” as I claimed . What the discussion highlighted for me was the importance of balance between the need to strive to design reusable services and the need to be pragmatic and fix service definition mismatches when they inevitably occur.

Anne Thomas Mane contributed to this discussion with points that I would at least partially agree with (A statement that she may find surprising as she identified herself as one of the "remaining hold-outs who continue to argue strongly that ESBs are irrelevant or transitory" who I referred to in the original posting.) Her points were:

1. “[An ESB] provides a means to expose legacy application functionality as services, and it supports mediation (although not efficiently) and sometimes orchestration”. Most ESBs provide good support for accessing existing applications, manipulating message formats and doing the work required to covert one or more existing systems into a black box service implementation which consumers can access without worrying about what is going on inside the box. I certainly agree with these statements.

2. “not all messages must flow through an ESB. Only those messages that require the services of the ESB should flow through the ESB.” To a degree, I agree with this one as there will be cases where the ESB provides little value beyond the acting as a common infrastructure through which all messages can flow – the value of which commonality must be judged on a case-by-case basis.

3. Anne’s third point is that by relying so strongly on an ESB – with its integration capabilities – it can undermine the focus on getting the service definitions right. In effect because an ESB can fix deficiencies with your service definitions on the fly, organizations can get away with poor service design – initially at least. Again I have sympathy with this but I feel it is like shooting the messenger. The SOA message is that the service definitions are crucial to getting the reuse/sharing which drives the economic argument. It is absolutely the case that adopting an ESB or any other product should not be used as a way of avoiding that architectural investment. Using an ESB without investing in service design is simply using an updated EAI product to do old style EAI!

Which brings me back to a favorite question: is it possible to create services which are capable of being used by all potential consumers now or in the future without some mediation (such as transformation orchestration) between the service implementation and consumer? I simply do not believe it is. Even if you can design the service to suit all possible needs today without mediation, there will be unforeseeable changes in the future.

The answer could be to refactor the service but the experience of many years of attempting to build large scale data models (which is effectively what you are trying to do – build a shared model between service definition and all the consumers) tells us that this gets more and more difficult to achieve as the scale (the number and diversity of clients in this case) increases. Therefore, the only alternative is to accept that to some degree there will be a need for mediation as the service definition will not be perfect for every consumer. The only remaining issue is whether individual ESB products provide sufficient mediation capabilities. And again I tend to agree with Anne when she criticizes the three ESB vendors that she name checks (IBM, Sonic and BEA) as these do not provide great mediation capabilities within their ESB products. However, current product deficiencies should not (yet) lead to the ESB baby being thrown out with the bath water.

Ronan

Posted by rbradley in Enterprise Service Bus • SOA concepts | Permalink | Comments (0) | TrackBacks (0)

2006 continued to be the year of the ESB in terms of new entrants and new products. However, the core problem with the category has not been resolved and in fact have got worse: Although every vendor seems to have one, nobody can agree on precisely what one is.

Throughout my tenure at PolarLake, I believed that (a) there was a need for complex integration middleware to implement a SOA – a view which now has become generally accepted (it certainly wasn’t the case back in 2001 when we started!). (b) the Enterprise Service Bus would be the title used for the complete stack of middleware required. It appears the industry is speaking or at least mumbling with different voices on this one.

Ron Schmelzer of Zapthink sums up the situation well when he says:

“The ESB craze has entered the final phase with JBoss entering the fray. The real problem is that despite all these vendors entering the market, there is even more confusion about what specific features an ESB must have.”

This makes the job harder for prospective buyers of ESB products in 2007. Right now the ESB tag can mean anything from reliable messaging with bells-and-whistles added, all the way across to what was formerly known as EAI!. As there is so much diversity, you need to dig deeper into more products prior to short-listing to see if the potential products are in fact comparable and could in fact address your problem.

The second issue is whether the glut of ESB badged products and vendors can continue and prosper. Both Steve Craggs and Computer Business Review in their 2007 predictions see trouble ahead for ESB vendors. CBR points out that there are already too many ESB vendors and with 4 significant Open Source plays (once JBoss appears), it will get harder for all the vendors and open source projects to stay the course:

“There are more vendors than the market will be able to support in the long term, and this will result in price pressure.”

Steve takes the next logical step and predicts:

“pure-play ESB vendors at least will struggle to be able to sustain a viable business model. On top of this, most of the large IT companies are either producing or have indicated the intention to produce an ESB. As such, pure-play vendors are likely to be squeezed out or acquired, and it is highly likely that at least some of this activity will happen in the next twelve months. “

(I would also include the Open Source projects in the danger zone as these are mostly single-vendor backed rather than relying on a broad developer community.)

Again buyer must beware – checking further into the long term viability of the vendor.

So am I saying stay clear of ESB in ’07? Definitely, not: If you are implementing SOA you will need the functionality provided by this motley crew of products – it is just going to take more time to work out which is the one for you.

Ronan

Posted by rbradley in Enterprise Service Bus • Market trends | Permalink | Comments (2) | TrackBacks (0)

SOA addresses such a broad problem domain that it is obvious that there must be significantly different architectural patterns or styles within SOA – each best suited to address different types of integration problem and within any organization it is likely that multiple styles will need to coexist. Understanding these distinctions in the context of your organization is clearly crucial to evaluating which products (or custom coded alternatives) should be used for a specific project. Equally, ensuring that the products selected for different SOA products work well together will be essential to ensuring SOA works across the organization.

Oracle published a white paper over the summer which described at a high level some of what I would call styles and they call “Value Patterns” emerging among their customers adopting SOA. These styles are:

• SOA based integration
• Modern, composite application development
• Modernising mainframe and legacy applications

The first two styles were also well described by Brenda Michelson back when she was with Patricia Seybold before launching out as Elemental Links Inc, in “Service Oriented World: Cheat Sheet” (Brenada calls SOA based integration “flow” style integration):

“The two primary styles of SOA used in business solution development are composite application development and flow. In composite applications, the user interaction drives a request for one or many services. Most of the service invocations are synchronous in nature. A composite application typically serves one business domain. Composite applications are often delivered in a portal.

In flow, business process and/or events drive the service invocations. The service invocations are a mix of asynchronous and synchronous; however, the overall flow is usually long running and asynchronous. A flow typically crosscuts business domains and often extends outside of the enterprise. "

(Oracle’s third style I am afraid I simply don’t buy as a different style: mainframes and legacy application modernisation may be the requirement but it isn’t a value pattern in itself. Although it does make sense to make the distinction at product selection as most vendors/products are either very mainframe focused or not at all.)

The style selected of course reflects the type of solution you are trying to create but it is often also influenced by the expertise within the organization which may reflect the problem domain or be an accident of history: if there is a strong background in EAI, flow will tend to be favored; if application servers are the preferred platforms, composite application development may feel more natural.

The three styles also drive very different technical requirements which strongly links back to product selection as products will tend to specialize in one style and provide weak(er) support for the others. This is of course at odds with the normal marketing hype of “Universal SOA platform for all known problems”.

Even in the case of Oracle, IBM et al, while their portfolio may include solutions for each style, it is very unlikely that a single product can address more than one style and the degree of integration between the products is then the issue. This is probably why Oracle stressed the level of integration within their latest releases as I previously commented on and why there is still a need to consider best of breed products unless the big vendor can really prove the strength of each product and their complete portfolio.

Posted by rbradley in Enterprise Service Bus • Market trends • SOA concepts | Permalink | Comments (2) | TrackBacks (0)

Within a couple of days of each other two very different organization starting to talk about the success they have achieved from moving to SOA: Wachovia, the 4th largest bank in the US and the District of Columbia.

A couple of year back, Wachovia embarked on what was understood to be a major investment and organizational shift – an investment strongly backed by the CEO of the corporate and investment banking division. It is also possibly the largest deployed SOA project in investment banking (that I am aware of at least). The remit reported in InfoWorld was to “Rock the boat, change everything, and build a multihundred million dollar, end-to-end, services-oriented development and delivery platform”.

Clearly, this was a project that followed the dream formula – the most senior sponsor signing up to stay the course. Also clear is that Wachovia believes that it is bearing fruit delivering more quickly and at much lower cost the types of custom apps that investment banks build all the time to support trading in new and ever more complex types of financial product.

In contrast, D.C. followed what is probably a more common approach – they had a serious organizational challenge that simply could not be easily fixed with the old point-to-point approach to integration: The city needed to address the crime rate and to do this the city agencies needed to work more closely together and this meant that they needed to integrate better.

Unfortunately, this integration required the 370+ data sources to be connected together. The solution was to make this the start of a SOA and implement based on an ESB (from SeeBeyond – now part of Sun):

“A few years ago, more and more [organizations] started using ESBs for data-sharing purposes,” Adam Rubinson, deputy CTO of D.C. said. Federal customers in particular have used this approach to unlock older databases.

The benefits were again clear and matched what other government agencies moving to SOA have found: It saved money in unexpected places – in this case because they were able to identify vacant properties and collect the higher taxes due on those properties. An important lesson for anybody migrating for SOA - regularly review you RoI model to ensure that you are picking up these unexpected benefits along your path to SOA.

D.C.'s experience echoes what I have seen happening in other US criminal justice organizations at state level and in the UK where the national Criminal Justice IT organization (CJIT) has been spearheading similar initiatives over the last few years.

For more details, the story is here for Wachovia and here for D.C.

Posted by rbradley in Enterprise Service Bus • Financial Services • Government | Permalink | Comments (0) | TrackBacks (0)

Forrester’s latest ESB study is already drawing the predictable press releases as each featured vendor promotes their own highlights. However, a more detailed reading reinforces a significant maturing of the products towards what I see as the eventual destination of ESB suites: matching the functionality of the BPM slanted integration suites from vendors such as Tibco and IBM (who of course are also repositioning these products as ESBs).

Before getting into the detail, Mike Gilpin and Ken Vollmer, the analysts who created this report, should be congratulated for creating what I would regard as the most detailed and coherent study from any analyst I am aware of. This is an update on their first report published back in December 2005, the creation of which I saw close up while I was still with PolarLake (one of the featured ESB vendors who I am delighted to see score highest in terms of current offering in this report).

Setting aside the actual detail of the report which can be read at the Cape Clear website, Forrester recognizes that most of the features previously identified with what Forrester calls integration-centric business process management suites or IC-BPMS (what others might simply call EAI suites) are now part of a fully featured ESB. And the newer features added to the IC-BPMS products (listed by Forrester to be Human Workflow, Simulation, Event Management and Complex Event Processing) will become available as part of an ESB by 2008.

To me at least this is not surprising: at the end of the day the capabilities required to solve complex integration challenges do not simply disappear with the move to SOA: We still need orchestration, data transformation and human workflow in the SOA world – not in necessarily precisely the same way as before, but we still need them.

So is an ESB just an updated IC-BPMS product (I really can’t see that acronym being widely used but I will stick to it as Forrester use it in the report)?

Absolutely not, the obvious benefits of being standards based – and hence reducing the cost of skilling-up and reducing the lock-in, and the lower license costs are fairly well known. And of course ESBs are typically built specifically to support Service Oriented Architectures.

However, given the need to justify any software purchase through Return on Investment, the most compelling difference that the report highlights is one that I have always felt is at the real heart of any ESB:

Customers use metadata to configure ESBs, which makes them easier to implement than IC-BPMS products.

Forrester’s research findings are clear on the benefits of configuration driven implementation: it halved the implementation time associated with ESBs over IC-BPM products. Knowing the methodology behind this report, this finding will be based on extensive customer interviews and not simply vendor marketing.

Posted by rbradley in Enterprise Service Bus | Permalink | Comments (0) | TrackBacks (0)