There has been a spate of customer success stories emerging over the last few weeks - one feature of these that is prominently highlighted is the focus on data. This should hardly be surprising given a major component of all integration problems (once you get beyond technology integration) is bridging between the data models. Within a SOA, the data models built are then exposed through the service definitions. The requesting applications must either formulate the request using the data model of the service provider or more commonly rely on an intermediary such as an ESB to bridge the gap (something I refer to as mediation).

Chief Application Architect Eric Peebles for the City of Chicago recently highlighted the role of data within SOA and he pointed out while speaking about why Chicago moved to SOA using iWay’s products:

city departments and even major citywide applications are separate islands of information about people, companies, land, and buildings. We needed a strategy to integrate these applications and data in order to drive business improvements more effectively.

There are two fundamental approaches to dealing with data integration within SOA: Build a global data model for your business which each connecting business unit must bridge into and out of in order to integrate with other units or build multiple data models. At their most extreme, there are major problems with each approach:

 Attempting to build a single global data model becomes harder as the size of the organization grows and the level of change increases. It requires an excellent understanding of the business at the start and an understanding of how change will impact upon it to successfully wire the data model for change. The approach of attempting to adopt a public standard is also not the solution (for instance FpML in financial services for derivatives) as there will significant divergences between the standard and what your business needs – it can be too complex, too simple or just plain wrong! That is not to say building a global data model is impossible – simply it is never as simple as it appears at the start!

 Allowing each unit to build their own data model and then providing bridging capabilities between each business unit. This builds cost and complexity into the integration layer which can slow down each project if the intermediary (such as an ESB) is not sufficiently powerful. It can also reduce the level of reuse between the business units of knowledge around building data models for your organization.

The solution is of course balance between the two approaches – and the precise balance will depend on the industry and the organization. For instance:

 If your architecture requires a central data exchange through which multiple organizations combine to complete business transactions, it may make sense to veer towards a global data model for this exchange.

 If your organization uses a lot of out sourcing or provides out-sourcing to other organizations, it may make sense to veer towards a per-unit data model as you will have limited control over the data model of the other parties.

Finally, the approach to building data models within a SOA context must be sensitive to the types of integration problems that need to be solved and specifically the data models that your model must integrate with. I have seen models which are theoretically perfect, but because of complexity and the mismatch between the model (exposed through service definitions) and what it needed to interact with where almost impossible to use. As with so much in integration, where possible keep it simple and build in the ability to change.

Posted by rbradley in SOA concepts | Permalink | Comments (3) | TrackBacks (2)

The discussions and news around SOA are clearly moving on from cool features towards tackling the nuts-and-bolts issues of how to get it done as can be seen from a few of the items that have appeared in the last week or so.

Joe McKendrick does an excellent job of pulling together a number of quite contradictory surveys attempting to measure exactly how great SOA adoption is today – with IDC coming in at the upper end with a survey claim that 90% of respondents were on the way to SOA.

His conclusion is that most organizations claiming to have deployed SOA are only in the early stages:

These statistics vary wildly, and lead one to conclude that SOA must be in the eye of the beholder, because one company's "SOA" may be, in someone else's view, a Spaghetti-Oriented Architecture or "JBOWS" (Just a bunch of Web services).

What all of the statistics show is that there is huge awareness of SOA and most people feel it is something that they should be doing. Clearly, there is absolutely no way that 90% of organizations or anything like it are already SOA based.

Accenture were recently reported as committing $450m to a new SOA technology lab in Chicago.
Meanwhile, Don Rippert, CTO of Accenture was in Hong Kong and outlined his view of the 4 phases of SOA development:

Phase one: "Organize and Strategize": get management buy-in and plan for SOA transformation

Phase two: "tactical implementations": embarkation on SOA projects and the conversion of applications into web-services, and the creation of business processes from such.

Phase Three: an emphasis on strategic and business services; consolidation of processes and services in creating an Enterprise Service Bus.

Phase Four: "SOA is industrialized," he said. This will involve cross enterprise processes, federation, predictive IT and business insight in real time.

Focused as you would expect on the high level, I am not sure there is much to argue with these statements. Each phase has its own challenges – in particular moving from phase 2 to phase 3 and beyond. That is where organizational and role changes need to begin. For instance, Wachovia took a product management based approach to ensure that SOA remains responsive to the needs of business.

And finally, David Linthicum has tackled the sensitive issue of whether the EAI dinosaurs can really dance to the SOA tune:

Many are finding that the more traditional *EAI technology is coming up short when you consider the new complexities of SOA.

And

Perhaps the core issue is innovation. While a commodity in the newer venture funding startups that don’t need to declare an EPS, are just cost to the larger companies, and thus to be avoided.

As a person who spent almost 5 years in a SOA start-up (PolarLake) I clearly believed and still believe that there is an opportunity for well-funded SOA start-ups to take on and beat the established vendors. However, some of the established vendors are already ‘evolving’ fast and their SOA offerings are already as impressive as the best of the start-ups. As so often happens in a new market, expect to see a couple of the familiar names disappear (as has already happened for instance to Mercator an old-reliable of the EAI marketplace) and a slot or two emerge for a new player.

Posted by rbradley in Market trends • SOA Predictions | Permalink | Comments (0) | TrackBacks (0)

Within a couple of days of each other two very different organization starting to talk about the success they have achieved from moving to SOA: Wachovia, the 4th largest bank in the US and the District of Columbia.

A couple of year back, Wachovia embarked on what was understood to be a major investment and organizational shift – an investment strongly backed by the CEO of the corporate and investment banking division. It is also possibly the largest deployed SOA project in investment banking (that I am aware of at least). The remit reported in InfoWorld was to “Rock the boat, change everything, and build a multihundred million dollar, end-to-end, services-oriented development and delivery platform”.

Clearly, this was a project that followed the dream formula – the most senior sponsor signing up to stay the course. Also clear is that Wachovia believes that it is bearing fruit delivering more quickly and at much lower cost the types of custom apps that investment banks build all the time to support trading in new and ever more complex types of financial product.

In contrast, D.C. followed what is probably a more common approach – they had a serious organizational challenge that simply could not be easily fixed with the old point-to-point approach to integration: The city needed to address the crime rate and to do this the city agencies needed to work more closely together and this meant that they needed to integrate better.

Unfortunately, this integration required the 370+ data sources to be connected together. The solution was to make this the start of a SOA and implement based on an ESB (from SeeBeyond – now part of Sun):

“A few years ago, more and more [organizations] started using ESBs for data-sharing purposes,” Adam Rubinson, deputy CTO of D.C. said. Federal customers in particular have used this approach to unlock older databases.

The benefits were again clear and matched what other government agencies moving to SOA have found: It saved money in unexpected places – in this case because they were able to identify vacant properties and collect the higher taxes due on those properties. An important lesson for anybody migrating for SOA - regularly review you RoI model to ensure that you are picking up these unexpected benefits along your path to SOA.

D.C.'s experience echoes what I have seen happening in other US criminal justice organizations at state level and in the UK where the national Criminal Justice IT organization (CJIT) has been spearheading similar initiatives over the last few years.

For more details, the story is here for Wachovia and here for D.C.

Posted by rbradley in Enterprise Service Bus • Financial Services • Government | Permalink | Comments (0) | TrackBacks (0)

One of the much touted benefits of SOA is that it aligns IT with the business – the concept of a misalignment between IT and business is one that I have always found puzzling: how much is just a cliché and when it is not, how can things have got misaligned in the first place. Clearly cliché or not, it is essential that SOA is seen to address this problem. To do this SOA must be implemented in any given company in a way that reflects the organization’s goals, structure, history and even its corporate culture.

Before telling you what my three principles are, lets go back to the perceived misalignment. As was noted by one blogger I read (unfortunately I can’t remember or find out who), how come nobody ever talks about how to align manufacturing with the business? It would be nonsensical – manufacturing can only exist if it is aligned. Similarly, it would be strange for accounting or finance to become misaligned with business. So how come IT can be misaligned? A few suggestions:

- There is a lack of credibility around IT in business – one measure of this is the number of CIOs without IT backgrounds. We need to accept that this lack of credibility is a reasonable response to a history of well-publicized overambitious and under-delivering projects.

- The IT structures in a corporation can become a power center independent of any business line – a cost center blamed for swallowing huge budgets without generating obvious business return.

- The scale of many IT challenge (particularly when it comes to integration) is often not appreciated by business people who do not understand why what appears straight forward can be hard and very expensive (what could be easier than linking a few databases!).

All of this predates SOA and as I mentioned a while back the NASCIO report made a key observation: one of the key aspects new about SOA was that it was building a justification for what has been best practice in enterprise architecture for a while: a justification targeted at business and hence at overcoming these issues.

Which brings us back to how to align your SOA strategy with business and some general principles that I have found along the way:

1. Adopt an incremental approach to rolling out SOA: Focus on achievable but worthwhile projects which can generate real and understandable benefits (such as NASCIO’s reference to collection of fines by a state agency which yielded rapid and measurable cash benefits). While SOA is all about architecture, don’t over-invest too early unless you are sure of the patience of your backers!

2. Don't attempt to use SOA to change the way the world is: Create your SOA strategy inline with the business organization as it is, not the way you wish it was - even if this makes your life more difficult. There is no point creating a single global registry with strict controls on who can add or amend items if the business is highly devolved and dynamic. Similarly, don't believe that your SOA will remove the complexity inherent within many business processes - it may remove technology complexity, but at the end of the day if the business is complex you will need a SOA that can handle that complexity.

3. The only worthwhile SOA is a pervasive SOA: The strategy has to be broad enough to allow most if not all projects to get going without excessive change-over costs or risks. Your SOA shouldn’t force major changes in infrastructure, otherwise the disruption could make the project too risky or the cost too high. Your SOA should be scalable down to the smallest projects in a meaningful way. If there are types of projects which can't be handled because they are too big or too small or too legacy focused or whatever, it will undermine the entire SOA program. All of whcih means that the technology platform(s) has to be plug-and-play to allow capabilities to be deployed where needed and stripped out for the projects which need a simpler solution.

Posted by rbradley in SOA concepts | Permalink | Comments (0) | TrackBacks (0)

SOA can appear overwhelming because as an architecture it is proposed as a solution to all integration challenges for every industry and for every scale and complexity of company. Clearly, the way SOA will be manifested in any given company will reflect that organization’s goals, structure, history and even its corporate culture. This is not only from a technology point of view but also from a business point of view – after all SOA is all about alignment of technology with the business.

The Forrester ESB wave research I mentioned in the last item reported different rates of take up among different sized firms, with 67% of larger firms implementing SOA this year compared to 44% of small to medium sized businesses. This is hardly surprising as the larger firms will typically have larger problems and larger IT departments capable and willing to take on SOA earlier in its evolution.

Aberdeen’s latest research focuses in part on a different issue: the emerging styles of SOA adoption. In particular, they identify three styles:

• SOA Lite is for users who are primarily deploying web services that do not require mission-critical capabilities such as high-volume scalability, high availability and failover, management, governance, and security.

• SOA ERP is used by companies that are choosing to deploy SOA surrounding their ERP application software.

• Enterprise SOA requires and uses mission-critical SOA middleware suite capabilities.

I don’t think that the fact there are different styles is in anyway surprising, nor are the three identified surprising as they match three common approaches to integration and within a single organization one or more will be present.

The first is for those organizations which solving light-weight problems in a direct and immediate fashion. SOA for these people provides a more systematic approach but must avoid over-burdening them with cost through over-engineering.

The second is the natural progression for organizations which have already chosen an ERP centric architecture – for these it makes sense to use SOA as an extension mechanism. It would make no sense to abandon the ERP centric world-view while moving to SOA.

The third is for those typically large organizations which have a multi-polar IT architecture, without a single center of gravity corresponding to an ERP system (for instance). These are typically the most complex organizations which probably have the most significant investment in middleware. This is the SOA style that gets talked and written about most – partly because these are often the organizations that adopted SOA first and they are also the organizations being targeted by vendors because they will make the largest purchases!

Of course, this is but one approach to defining SOA styles. You could also think about vertical specializations of SOA as David Linthicum recently covered or more technical distinctions between event-focused SOA and RPC-focused SOA, a distinction that unfortunately spawned the concept of SOA 2.0 (and I use the word spawned deliberately).

The inportance of all of this is that it allows us to move from the view of SOA as all encompassing (and confusing) architecture to something more focused on our problem domain – hence allowing appropriate architecture and product selections to be made. The fact that styles are being thought about and identified now simply shows the maturing of the SOA world.

Posted by rbradley in SOA concepts | Permalink | Comments (0) | TrackBacks (0)

Forrester’s latest ESB study is already drawing the predictable press releases as each featured vendor promotes their own highlights. However, a more detailed reading reinforces a significant maturing of the products towards what I see as the eventual destination of ESB suites: matching the functionality of the BPM slanted integration suites from vendors such as Tibco and IBM (who of course are also repositioning these products as ESBs).

Before getting into the detail, Mike Gilpin and Ken Vollmer, the analysts who created this report, should be congratulated for creating what I would regard as the most detailed and coherent study from any analyst I am aware of. This is an update on their first report published back in December 2005, the creation of which I saw close up while I was still with PolarLake (one of the featured ESB vendors who I am delighted to see score highest in terms of current offering in this report).

Setting aside the actual detail of the report which can be read at the Cape Clear website, Forrester recognizes that most of the features previously identified with what Forrester calls integration-centric business process management suites or IC-BPMS (what others might simply call EAI suites) are now part of a fully featured ESB. And the newer features added to the IC-BPMS products (listed by Forrester to be Human Workflow, Simulation, Event Management and Complex Event Processing) will become available as part of an ESB by 2008.

To me at least this is not surprising: at the end of the day the capabilities required to solve complex integration challenges do not simply disappear with the move to SOA: We still need orchestration, data transformation and human workflow in the SOA world – not in necessarily precisely the same way as before, but we still need them.

So is an ESB just an updated IC-BPMS product (I really can’t see that acronym being widely used but I will stick to it as Forrester use it in the report)?

Absolutely not, the obvious benefits of being standards based – and hence reducing the cost of skilling-up and reducing the lock-in, and the lower license costs are fairly well known. And of course ESBs are typically built specifically to support Service Oriented Architectures.

However, given the need to justify any software purchase through Return on Investment, the most compelling difference that the report highlights is one that I have always felt is at the real heart of any ESB:

Customers use metadata to configure ESBs, which makes them easier to implement than IC-BPMS products.

Forrester’s research findings are clear on the benefits of configuration driven implementation: it halved the implementation time associated with ESBs over IC-BPM products. Knowing the methodology behind this report, this finding will be based on extensive customer interviews and not simply vendor marketing.

Posted by rbradley in Enterprise Service Bus | Permalink | Comments (0) | TrackBacks (0)

Sometimes reading about Web2.0, the evangelicalism obscures what could be useful for the enterprise. No better example of this than the quote from David Girourard of Google who says:

Enterprise software is entirely bereft of soul. It is designed for business not for humans.

More analytically, Barry Briggs defines the difference between SOA and Web 2.0 as such

SOA is heavyweight but robust enterprise architecture; Web 2.0 is democratic, social and participant-based. On the surface, they are orthogonal. They target different problems: so it's just as hard to imagine building a lightweight wiki with SOA as it is a global supply chain with (say) MySpace.

I won’t get into the argument about whether SOA as an enterprise architecture must be inherently heavyweight or whether it is self-imposed complexity – neither position I agree or even disagree with wholly. Instead what I found interesting was the definition of Web2.0 as democratic, social and participant-based.

This idea is expressed in a slightly different way in Don Hinchcliffe’s web2.0 blog:

Web 2.0 emphasizes a social aspect that SOA is completely missing. And probably to its lasting detriment. SOA has much more central control, management, and governance while Web 2.0 is free wheeling, decentralized, grassroots, and with absolutely no command and control structure.

Clearly, the emphasis of Web2.0 is user interaction rather than system to system interaction as I covered in my last blog item in this series, But every SOA deployment must have users and hence there must be ways to leverage Web 2.0 into a SOA environment.

More importantly from blogs to wikis to sites such as bebo and MySpace, Web 2.0 shows us that it is possible to engage individuals who are not primarily techies and actually get them to use less polished software solutions in exchange for more control. Anybody who has edited wikipedia entries will know that the user interface isn’t polished – but frankly it is "Good Enough" and “Good Enough” is the key measure when the user actually wants to use a tool because they see the benefit.

The question is how does this translate into the enterprise – the enterprise is of course full of people who use computers but are not computer programmers. While not computer programmers, these users are willing to use relatively difficult tools (and excel is the perfect example that I mentioned in the last item) if they are allowed to.

The remaining issue then becomes will they use Web 2.0 tools and even should they be let (or should that privilege be kept exclusively for professional computer programmers) . This is question raised by Tony Baer in a comment to the last item in this series

Given the loosey goosey nature of Ajax tooling and practices at this point, it will be interesting if we see in Web 2.0 a replay of the client/server era when VB gave non programmers enough ammo to be dangerous.

Why did VB proved such a “dangerous tool”? If we agree to set aside immaturity in the tool at the time Tony probably refers to which made it easy to do the first 60% and really hard to do the last 40%, I would suggest that the major problem was that the corporate IT environment lacked in governance: it relied on the fact that the ‘citizens’ living on the edge couldn’t do much unless allowed to do by corporate IT and hence ‘citizens’ were well-behaved because they didn’t have much choice in the matter. These new VB based citizens (mostly accidently) didn’t stick to the unwritten and unenforced rules and caused problems.

However, SOA recognizes that it is no longer possible to maintain such a rigid environment if we want to encourage reuse and reflect the new business models from process outsourcing to software as a service. The SOA blueprint is to create an environment with not only well defined services but also good governance which maintains and ensures "good" behavior. Because of this, within a SOA environment there should be much greater potential to devolve more control to the users: If the account wants a mash-up showing latest invoices and inventory – let him build it. If the sales team wants to track key accounts in the pipeline against press releases on yahoo – let them build it. The governance structures will stop unauthorized access and will spot unacceptable grabbing of bandwidth or breaches of security and so on.

So in theory we are set for an explosion of use of Web2.0 technologies in the enterprise. In reality, I see three blockers along the way:

1.The tools are mostly still designed by techies for techies and what a techie finds acceptable is far from what a business user will use. Therefore, we need to identify the tools that are mature – or drive the maturity - and build projects around those.

2. Can the user community motivate themselves to go and do it? This will vary from organization to organization: It also requires some more thought about how these new tools will be used by non-IT specialists and what they will do. For instance, we can see excitement growing about building Business Intelligence mash-ups. Is that really the big win? I am not sure.

3. Will corporate IT give up its habits of control and let go? History isn’t great on this – the PC revolution came from the departments and was not understood or even resisted by central IT in many cases. Corporate IT needs to recognize that this is an opportunity for a significant rebalancing away from its role as provider and towards its role as policy setter and enforcer.

Posted by rbradley in | Permalink | Comments (1) | TrackBacks (0)