We use cookies and other similar technologies (Cookies) to enhance your experience and to provide you with relevant content and ads. By using our website, you are agreeing to the use of Cookies. You can change your settings at any time. Cookie Policy.
Start a Discussion

What's the Best Way to Measure SOA Success?

Vote 0 Votes

From Joe McKendrick: There's been a lot of discussion on the metrics that need to be applied to SOA-based projects and services to determine their value to the business. But companies seem to be having a hard time figuring out ways to determine if their SOA is delivering. What's the best way to see if your SOA project is delivering or is falling short?

10 Replies

| Add a Reply
  • This is a very difficult question to answer in this forum. I gave an entire presentation on this subject recently at the SOA Symposium in Washington DC. I will attempt to summarize my findings here in a couple of paragraphs.

    1. SOA Success is contextual - that is, it is based on your goals. Specifically, are your goals long-term agility, cost reduction or better IT/Business alignment? A successful SOA engagement need not be successful in all these missions.

    2. What is your intent for your SOA engagement? Proof-of-Concept? Re-engineering? Semantic mapping? Stating the intent up front and then being able to illustrate how the money, time and resources used for the SOA engagement directly support the intent = success.

    3. SOA success need not be deterministic. Changing paths midstream based on what you have learned is acceptable. What is deemed success may merely be a stepping stone in a much larger requirement that will only be uncovered once the first step is completed.

    Discussions of ROI and other quantitative results can be applied to subsets of an SOA initiative, but the success on the whole is going to be highly dependent on qualitative factors.

  • There's not much I can add to JP's excellent answer, so allow me to reiterate one of his points. You define your success against the goals you've set. It is absolutely critical to define clear goals at the beginning of a SOA initiative. The biggest cause of SOA failure is when stakeholders do not articulate what it is they are trying to achieve.

    SOA can only be a success if you know what success means to you.

  • First, the wrong way to measure overall SOA success: Counting reuse. High or low reuse doesn't matter so much as having the right reuse. Better to measure the business value of the solutions delivered using SOA, especially noting which of the solutions would not have been possible without SOA. Next best is to measure SOA processes: Are services being designed well? Are the right cross-functional teams involved in planning the future of the SOA business service portfolio? For a new project, how many of the services it needs already exist, and is this number trending upward (at least for projects in the same general business domain)? Are solutions being delivered on the strategic SOA infrastructure, or is each project deciding its own divergent infrastructure? These types of metrics keep SOA initiatives focused on the business and on the efficiency and effectiveness of SOA-based solutions, both of which indicate overall success with SOA.

  • Metrics are clearly important, but for me the real answer to your question is pretty simple. Did you solve the business problems you set out to solve, and did you do so in a way that was more efficient or effective than before? Further, will the solution scale as your business grows.

    If you create an incredibly elegant solution that functions brilliantly from the technology standpoint, but does nothing to solve any of the business issues it was supposed to address,then that's not success.

    • user-pic

      Here! Here! The most succinct definition I've seen. Too bad more techies/enterprise architects do not understand this concept.......

  • Although the omnipresent "it depends" answer is the most correct, I see some trends towards measurements that help drive success rather than measurements OF success.

    Of course measurements OF success help to drive success as well as these metrics give people hope and of course proof of the value of their SOA efforts.

    But I feel that measurements that help to drive success are the best way to align the "tribal" behavior of organization. For example, one of the aligning concepts that gets measured is one that I call "total cost of behavior across the lifecycle".

    This measurement is a way of capturing the downstream cost, risk and complexity of any behavior. By "closing the loop" you can link these downstream costs of behavior to chargebacks and other behavioral incentives thus nipping the behavioral root of any externalities in the bud, so to speak.

    This is very helpful in aligning different "lifecycle tribes" such as developers, IT operations, QA and such.

  • The Free Dictionary defines success as "the achievement of something desired, planned, or attempted." Success in an SOA initiative follows the same definition. SOA, in its purest form, is meant to align business and IT towards a utopian agilily. Real-world SOA projects typcially have more short term goals in mind such as reusability, maintainibily, interoperability, etc. Regardless, the success criteria of any SOA initiative should be defined upfront based on the goals of that initiative. It goes without saying that proper care should be taken that all goals and success critera are SMART and agreed to beforehand by all stakeholder in unambiguous terms. Ultimately, just like beauty, I think success, too, is in the eyes of the beholder.

  • Great answers all...

    One more thought.. in creating a strategy for measuring SOA success, success will mean very different things to different stakeholders. It is key right up front to identify who the stakeholders are and what matters most to them (and this will also help identify right up front if some success expectations are orthogonal or "at odds") A simple example....

    SOA success to the VP of applications responsible for the implementation of apps that support the "mobile services" business unit in a telco may be measured by metrics such as reducing time to provision new services to customers and increasing customer retention. SOA success to the VP of application operations may mean reducing application outtages and SLA violations and better capacity planning. SOA Success to the head of Enterprise Architecture may be measured in reducing the total cost of application change from request to deployment and increasing interoperability policy compliance.

    This is not to represent that these are the "right" success metrics but rather to make the point that success isn't a one size fits all measurement and especially so with something as potentially transformational as SOA. Success will have a meaning to business, to applications, to development teams, to quality, to operations and so forth. The key is to make sure the goals are clear, supported, measureable and not in conflict. For instance, if Enterprise Architecture has a goal to increase the availability and visibility of shared services yet development teams remain measured on speed of implementation and reduced dependencies, these would be at odds and the organization would have to rationalize them via collaboration and negotiation. (and some potentially uncomfortable cultural changes)

    To sum up, the key to success in measuring SOA starts with identifying what success means at all levels of stakeholders in the organization, it cannot be "one size fits all".

  • user-pic

    As the stated benefit of applying SOA is increased business agility this should also be the argument used in describing SOA's success! In what way did agility increase (KAI Key Agility Indicator(s))? What is the business value of that increase and what was the investment? Please check this article on this subject.

  • I largely agree with the replies above. As SOA is gradually becoming the reference architecture, I propose however to measuring the success of SOA by calculating the cost of no SOA, the CONI (Cost of No Investment). It can be the cost of lost market share due to insufficient agility, the cost of compliance violations, the cost of a change in process, etc.

    Mathematically, it makes obviously no difference. Psychologically, however, the term CONI might give some extra motivation to speed things up.

Add a Reply

Recently Commented On

Monthly Archives