Of course in the limited time available in a live panel, we aren’t going to try to boil the ocean. Instead we are going to discuss how SOA investments can be augmented to address the currently particularly hot issue of visibility and control. However, with the launch of ebizq’s financial services industry solutions tab , I hope and expect that this will be the first of many finance specific event on ebizq. On that basis, let us know the areas that are of interest to your organisation and your perspectives on the latest trends in FS IT.
Ronan
“SMash addresses a key part of the browser mashup security issue by keeping code and data from each of the sources separated, while allowing controlled sharing of the data through a secure communication channel.”
The possibility of leakage between sources is a key issue not only from a security perspective but also from an Intellectual Property perspective (the content may be licensed for specific use and this sharing may incur additional fees). I look forward to discovering if IBM’s proposal manages to balance the need for security against the need for usability. While this is the classic trade off with all security, it is particularly acute for Mash-ups which have the ease of creation at the heart of the value proposition.
Ronan
The Ilog announcement is interesting in that it clearly states two of the most common requirements in Insurance that SOA (and in this case SOA with additional business rules) tries to meet - and one which is less commonly stated:
“First, Hiscox wanted to be able to add new distribution channels as quickly as possible.
Second, the company wanted to reduce the time and cost associated with both making changes to existing products and bringing new products to the market.
Finally, Hiscox needed to increase the ability of underwriters and business analysts to make changes to rules directly without having to change complex system logic, allowing the company to improve its business response time.”
The ability to add new distribution channels is common across financial services. As the products are purely electronic, adding a distributor is equivalent to integrating their systems with your own. This requires mediation of the message formats and flows – clearly Hiscox is using a rules based approach but of course others are possible.
The need to get to market quickly with novel products is also common across insurance, retail banking and its most extreme case in derivative trading. Again, it requires good message manipulation capabilities in the middleware.
The final requirement is to my mind the most controversial and clearly the most valuable if achieved: To take IT out of the loop when changes need to be made. If it can be done, it has clear cost and agility benefits. However to be successful it requires well formulated message formats, well understood processes and above all careful controls around what can be changed as the underwriters and business analysts are unlikely to appreciate the implications of their actions in the way IT staff have been long trained to do.
Ronan
]]>"you can't create a service that is designed to support 100 trades per minute and then let someone with a black-box trading system connect to it and expect happy results. The resulting automated traffic would bring the service to its knees and those supporting it would be faced with fielding phone calls from users letting them know the system had gone down."
However, system outage is only one problem - you must also ensure that mash-up user is entitled to use the data (both from an authorization perspective and also from a rights-management perspective as the data may not be licensed for use by the mash-up user).
The only solution is to increase the level of control and governance by defining and enforcing SLA and so on - in effect bringing mash-ups back with the framework of enterprise IT. The challenges will then be
- Finding out where mash-up development is happening in the organization. The very ease with which mash-ups can be created will tend to put them under the radar.
- Convincing the users and developers of the mash-ups that governance is there for good business reasons and not simply central IT attempting to regain control and squash innovation that they do not control.
Ronan
]]>The business needs associated with this data deluge are not only storage – and in some cases storage is not relevant as the data has only transitory value. The greater value is in identifying the anomalies or unusual combinations that may correspond to the financial opportunity or a potential fraud. Oracle is clearly trying to address both the traditional transactional processing requirements and these new requirements which are more usually solved with Complex Event Processing tools.
As Oracle’s Dave Chappell puts it, XTP allows “transactions to occur in memory and not against the backend systems directly due to the need for extremely fast response rates, but still including transactional integrity” Fair enough.
However, he goes on to say something that Streambase and Apama (two CEP engine vendors) among others may be surprised to hear:
“Usually, a complex event processing engine is something that can also capture, correlate and apply decision rules to look for specific patterns in events over time and look for exceptions. However, there are certain applications that operate on streams of event data that is so voluminous that typical backend solutions or storage solutions just can't handle it.”
The whole purpose of CEP engines is to provide real-time analysis of massive event streams. My understanding is that to do this they precisely by avoiding using the backend solutions which Dave refers to. Which makes Dave's remarks rather confusing.
Stepping back however, it is clear that this announcement further demonstrates the growing interest among vendors in the potential value of the data deluge problem being solved. Whether the demand for these solutions will grow and break out of its traditional niches (algorithmic trading in financial services and some specialised security applications) is not yet clear.
Ronan
The announcement from Abode about the latest version of its RIA platform reminded me that the issues are not only associated with the ‘back-end’. In particular, the RIA space is developing fast but is still immature. The world has certainly moved well beyond the appearance of Google maps and AJAX. (Of course, AJAX was always a confusing term in that it gave the impression to outsiders that there was a well-defined piece of software rather than a general approach to creating RIA.) However, the RIA space remains an incredibly busy space as can be seen from the list of 25 different RIA frameworks listed here . This diversity reflects a great level of innovation which is good in the long term. And it is fair to point out that RIA is a wide definition – some of the 25 are focused on specific types of applications such as the Ruby on Rails for developing database backed web applications while others such as Tibco’s General Interface are clearly intended for use with enterprise middleware at the back.
This poses the challenge for the enterprise architect of which horse or horses to back. The easy answer that I have heard is that it doesn’t matter: RIA is supposedly easy to replace and are often used to create opportunistic applications (quick to create with short life-spans). If you change your mind (the story goes), simply rip and replace with another RIA. Equally, (the story continues) it doesn’t matter if one department goes for one RIA and another a second.
I remain unconvinced. Yes it is true that it is easier to replace an RIA than to replace infrastructure software such as message queues. To stretch an analogy, if switching message queuing is like a lung transplant, then switching RIA might be grafting back a finger – while I definitely wouldn’t like the first, I wouldn’t volunteer for the second either. What is being ignored is the cost of replacing or maintaining in parallel RIAs based on old tool-kits and new. What is also being ignored is that the users of these applications are the sensitive business users who will not appreciate the disruption of their favorite tool which follows on from switches in strategy.
All of which means that we need to look at the RIA options with a great deal of care: The opportunities and benefits are certainly considerable. However as with other bets on early markets, there are significant downsides as we inevitably make the wrong choices from time to time.
Ronan
Most of the OSS SOA solutions have up until recently focused on the base infrastructure required for first generation SOA projects (I say most as Bosteq for one took a different tack by focusing on the major 'higher level' problem: developer productivity). What has been lacking or weak with the OSS offerings has been the enterprise level capabilities such as management and application connectivity which are essential for any widescale deployment of enterprise infrastructure software.
Alex Fletcher's blog on RedHat's ambitions for its SOA strategy highlights two recent partnership announcements which shows RedHat is the latest OSS SOA vendor to start to address these gaps: Partnering with iWay on enterprise application connectors and with Hyperic for management.
iWay has become the standard port of call for most if not all SOA vendors needing an enterprise application connector story - I say story because while end-users like to know that such connectors are available, they do not necessarily end-up buying them. For RedHat, it presents some interesting issues: iWay is not an OSS vendor - and hence the customer is forced into a hybrid model of OSS and closed source. While this may not be a problem (IONA follows a similar model within its own family of closed and open source products), it seems to be a major disparture for a company such as RedHat so closely aligned with OSS.
The Hyperic relationship is also a little me-too as it follows the path already followed by MuleSource. In this case there are no 'religious differences' as Hyperic is also OSS. And me-too has some obvious advantages for the users: It is better if vendors converge onto one solution as any widescale SOA deployment is bound to include multiple SOA platforms which need to be managed as a whole.
Which makes both announcements positive steps for RedHat. However, they still have a long way to go before they are ready to win the 50% of the market which Alex points out is their strategic goal. And in my opinion they must greatly accelerate their moves to make their platform enterprise grade if they are to have any chance to achieve that goal by 2012 as is their stated aim.
Ronan
]]>Over the last 3-4 years in particular, Open Source has been one of the major feature of SOA technology landscape and looks like only becoming stronger as OSS projects mature and IT budgets get squeezed in the economic downturn. Denis Byron also celebrates the anniversary and provides some excellent counterpoints to the more outlandish OSS claims in general. To quote Dennis: " I can't quickly think of any (never mind "many") "business computing category" in which open source software is a leader". In the context of SOA, OSS is a major feature - although it is not a leader as yet.
XML is also 10 years old - I almost felt like writing 'only' as XML is so much part of the fabric these days it is hard to imagine a time before it existed. Over the 10 years, XML turned out to be a giant leap forward by allowing data formats to be defined independently of technology in a way not attempted before and launched countless bodies to define XML standards for industries from telecoms to financial services to travel.
In 'celebration', Tim Bray published a long piece on the people and early history which Tim Anderson both summarises and criticises for Tim Bray's hostility towards Microsoft (a common feature of the OSS community as well of course). Criticism is hardly surprising when Tim Bray makes the delightful remark that "Mick [Microsoft in his semi-allegorical history] is a domineering, ruthless, greedy, egotistical, self-centered, paranoid bastard.".
Tim's views on Mircosoft's attitude to standards are clearly heart felt. However, having been involved with standards processes over the years - I am not sure Microsoft is necessarily that much more 'evil' than other companies which attempt to manouver the standards process to aid their commercial goals. Also having lived through the CORBA/DCOM wars in the 1990s when Microsoft was not part of the process, it seems clear to me that XML benefitted more from Microsoft's involvement more than it would have benefited from being pure and free from Microsoft which no doubt would have followed a different track and created a competing standard.
Ronan
Specifically, some vendors and analysts focus primarily on the software tooling such as registries and repositories (and even somewhat surprisingly on management). Most end-users, excepting those recently indoctrinated, interpret it to mean the definition and 'human' enforcement of policies associated with effective SOA roll-out and maintenance. This is hardly surprising – end-users assume that SOA Governance is going to be related to governance first and enabling tools secondly. While SOA does present significantly different governance challenges, it starts in the same place as 'traditional' governance: definition of governance policies and processes for creating and evolving these policies. Todd Biske seems to of a similar mind-set when he writes in his excellent blog:
“I’ve said it before, and I’ll say it again. Governance is about people, policies, and process. Tooling really only comes into play when you start looking at the process portion of the equation. I don’t want to dismiss tooling, because it absolutely is an important part of the governance equation, but if you don’t have the people or the policies, tools won’t help.”
If you wonder why I am getting hot under to collar on this one, this is why: The focus on SOA Governance tools does not help most organizations deliver on SOA governance. In fact it is a distraction as the best tools in the world cannot deliver the organizational and cultural changes necessary and may lull the organization into a false sense of complacency.
Ronan
“IBM wants to take the technical allusions out of the term itself, referring to it as “Business Event Processing.” The renaming to BEP makes sense, since I doubt if a concept that starts with the word “complex” is going to win a lot of converts.”
However, I don’t disagree with sentiment that CEP could be the next big thing. It has been considered and used a little to deal with the general increase of data volumes in some industries (financial services being in the “lead” as I covered here ).
However, there is potentially much more opportunity in the SOA context as IBM seem to recognize. As the volume of network traffic related to SOA increases, the need increases to monitor, manage and react to anomalies. Anomalies which could correspond to simple human error, to unexpected loads and even to illegal or fraudulent behaviour. Detecting an anomaly may require looking into an event, detecting specific sequences of events or any arbitary combinations of the above. CEP provides a good approach to detecting subtle anomalies occurring in massive message flows in real-time. While you could have used CEP (if it had existed) in the pre-SOA world, it would not have made much business sense: SOA potentially divides many more business processes between applications (into the SOA layer) and also increases the total volume of messages. The combination makes the total value of the messages in the SOA layer significant and therefore controlling the SOA layer becomes as important as managing applications.
The caveat is how quickly and in how many organisations if at all will this level of SOA deployment occur. I remember talking to a senior architect in a US bank last year who was pro-SOA but would never consider shifting the centre of gravity of business processes out of applications. Hence he would never need the sort of CEP control layer capability that I was proposing.
Time will tell – as with most “next big things”, it will take longer and be less massive but CEP will certainly be discussed and used in increasing levels throughout 2008 and beyond. And no, I don't think many people will call it BEP any time soon.
Ronan
]]>“SOA proponents need to redouble efforts to emphasize the business case, and de-emphasize the technology aspect, he said. "It all boils down to architecture." Still, he said, the reuse inherent in SOA can be made to work, and projects where CEOs can see demonstrable value -- such as a real-time analytics dashboard -- can achieve quick success.”
Meanwhile David’s ZapThink colleague, Jason Bloomberg told SearchSOA that
"Furthermore, enterprise mashups are becoming the killer use-case for SOA, that is, the ostensible reason for doing SOA from the perspective of the business. So, SOA is stronger than ever, it's just becoming part of the woodwork. Enterprise mashups are the part that shows."
I think that for better or worse these quotes reflect the reality. SOA is not an easy proposition to sell to business on its own – it is about what you use SOA to do. SOA is certainly a better way of doing internal application integration but these traditional application integration projects are not particularly interesting to the business sponsors. Lack of interest means that cost is everything and SOA can be a very hard sell as early projects have to bear the brunt of the SOA set-up costs (called the SOA tax by some). Therefore, SOA is more likely to flourish in areas where the additional initial overhead can be justified by greater returns and more importantly returns that are of greater interest to the business.
The concept of an enterprise mash-up has been attracting interest outside of the IT department. The concept is inherently attractive to business users as it suggests that there is a way to easily combine the information the business user cares about without apparent interference of IT departments with their long and expensive project cycles. Potential applications from CRM to executive dashboards quickly jump to mind.
Unfortunately, there remains no such thing as a free lunch. Enterprise mash-ups rely on the availability of reliable mechanisms to access the data and suitable access controls to ensure that only authorised users get to what could be very sensitive information. While it is possible to roll out pilot mash-ups, over time proliferation of mash-ups will have a much greater impact than the casual observer might expect: Each mash-up user touches potentially many systems and mash-ups often rely on directly polling the system at regular intervals. This means that large scale roll-out of mash-ups will put large and potentially unexpected pressure onto the applications and infrastructure if not carefully controlled.
Which brings us back to SOA: Large scale deployment of mash-ups will require enterprise grade integration infrastructure as provided by SOA. The challenge for IT departments will be to explain to the irate business user that the reason why their mash-ups don’t work so well any more is because they have to invest more in those vague and intangible concepts: middleware and architecture. We may arrive back to square one but at least our business colleagues may better understand why we need to be there!
Ronan
]]>“What ZapThink is finding is that the primary barriers to SOA adoption do not come from business management, which by and large realize the benefits of an agile, reusable, and loosely coupled architecture (even if they don’t call it that), but rather from within the IT organization that resists the movement to SOA for a wide range of reasons”
And Ron certainly makes some good observations on why IT departments can struggle with adopting SOA – everything from the challenge of making the required organizational change to general fear of the new to believing SOA could threaten their jobs.
While it is obvious as Joe McKendrick points out that "few analysts, commentators or pundits seem to argue the point that SOA is facing resistence", I think it is too simplistic to say business gets it and IT doesn’t or doesn’t want to. First of all, lets be realistic: Of course SOA is facing “resistance” – any major shift, as SOA is, takes time to be adopted and some organizations will go slow, others will jump ahead. Frankly - either decision can be right depending on individual circumstance.
Secondly, because SOA promises benefits that no sane business person would turn down does not mean that every business person will drive it though the organization (“the business sell” as David Linthicum puts it). What I suspect ZapThink is picking up from some SOA proponents is a “soft yes” to SOA from business, based on the benefits and frustration when IT management can’t convert this to a “hard yes” with budget. The reasons for this difficulty have been widely discussed and relate to the generality of the benefits which can be SOA’s worst enemy as my colleague Steve Craggs puts it:
the platitudes surrounding SOA have been used for 20 years on business execs, so they can be forgiven for having grown rather cautious, and desirous of looking for incremental investment where payback can be validated at each stage.Ronan ]]>
In the past (pre-SOA), I have been sceptical about industry specific frameworks and in particular wondered whether they were actually clever ways of packaging up consultants with expensive industry specific knowledge. A scepticism I brought up with a senior manager in a European retail bank who is an early adopter of SOA of about 4 years standing. His view was that leveraging the industry based patterns that he is now using (provided by IBM in his case) earlier on would have been a significant benefit in reducing the effort and risk associated with his programme. As such a framework was not available when he started, he had to create his own framework: Using somebody else’s even if it had to be adapted would have been a major benefit in terms of time, cost and risk.
Of course whether it will work for you will depend very much on the industry you operate in, whether its processes are sufficiently standardised and whether it has attracted the attention of one of the firms creating the frameworks. However for industries where it does suit the benefit for users will be significant and the opportunity for vendors will also be big. In fact, I believe that for these “framework-friendly” segments the competitive landscape may be considerably different: The vendors with the greatest knowledge of providing industry specific solutions will be in a very strong position. In particular IBM and SAP should dominate as they have the customer base and expertise and are already investing heavily on creating the frameworks.
As such it may well provide SAP will an excellent opportunity to break out of its ERP-centric SOA niche (selling to customers who see SOA as an ERP extension) and get some return on the investment in their Cinderella-like SOA strategy: just like Cinderella, it does all the right things but still gets ignored by all the Prince Charmings!
Ronan
Rather than going into detail on what EDA is, I will quote some definitions from Jason Bloomberg of ZapThink back in 2004 and suggest that anybody who wants to know more goes to Brenda Michelson’s excellent primer on the world of events and EDA in her elemental links blog.
Jason’s definitions are:
EDA is an approach where events trigger asynchronous messages that are then sent between independent software components that are completely unaware of each other.
While
SOA, on the other hand, is an architectural approach where software functionality is exposed as loosely coupled, location independent services on the network.
Therefore the primary difference between SOA and EDA is the level of decoupling between the software components. The total decoupling of source of the event and the eventual destinations in EDA means that the message fully defines the interaction between components – a very different situation from SOA where the server and client interaction is ruled by the service definition. In fact, EDA is not a new pattern – it is a form of “publish and subscribe” which has been around for a long while and effectively used to solve certain classes of problems.
The question from Joe was “Is EDA the new SOA?” – I would say no for two reasons:
• The level of enforced decoupling in EDA can make it awkward to shoehorn the range of problems that an enterprise architecture has to solve into a pure EDA form.
• For those problem types that EDA is well-suited for, SOA can be extended with a bit of EDA and therefore do the job.
Rather, I see EDA being used ‘on top’ of SOA – to allow identification and processing of unusual events or combinations of events that should generate alerts or recovery processes. SAP for one is already providing some of this type of capability within its NetWeaver product set where BPEL-defined processes can be fired off in response to specified events. This type of functionality will be crucial in terms of delivering control across the SOA-based network of integrated components exchanging more information and information of greater business value.
Ronan
]]>Therefore, I noticed with some initial dismay the appearance of the SOA consortium . However after finding out a little more about it, I believe that the SOA consortium has the potential to be a useful addition to the world of industry groups for a number of reasons:
- It is focused on advocacy and not becoming yet another standards body: SOA needs effective supporters, not more technical complexity right now!
- It recognises not only the importance of engaging with the business but also the need for both business and IT management to change in order to benefit from SOA.
- Its founders include not only major vendors (IBM, HP, BEA, Cisco and SAP) but also major end-users (Bank of America and Avis). This balance between vendor and buyer is essential if it is to become credible as an advocacy group.
- Finally, it also has the involvement of two industry organizations who know how to engage both end-users and vendors in a constructive way. In particular, the OMG has a long track record in doing this successfully. (The Integration Consortium is a mere baby in comparison, but set-up with the same concept of end-user engagement).
My only caveat is that the claim to target the Global 1000 seems undermined by the consortium’s decision to hold only three meetings to formulate its charter – in the obvious locations of New York, Dallas and San Francisco. While there must be limits to consultation, this choice does seem a little short-sighted if the consortium intends to be truly global.
Ronan