It's almost a truism that cultural and organizational factors -- these include politics, information silos, "tribal" interests, and effective change management -- generally determine the success or failure of a major IT initiative in most organizations today. While "big bang" strategic projects and infrastructure upgrades are notoriously fraught with peril, particularly when it comes to creating real bottom-line business impact, there are few IT subjects that hit upon the issues above more directly than the practice of service-oriented architecture (SOA).
Theories abound to why this is. One reason is that SOA's vision of enterprise-wide interoperability, open data sharing, and a pliable strategic business architecture is often at odds with the present state of the business. The future state of a SOA vision is frequently misaligned with current stakeholder motivations and existing technology portfolios.
In other cases it's simply that technology change moves at light speed compared to the seemingly glacial pace of local business culture and organizational evolution. Maybe, as is sometime prescribed, getting people to look at their business in a fundamentally different way is the key. But this then makes SOA success a "soft" skill at which IT is notoriously poor.
What then is to be done when it comes to achieving SOA adoption and driving successful governance long-term?
SOA works, but how we do it often doesn't
These days the conventional wisdom of the SOA community has generally arrived at the conclusion that the concepts themselves are sound, but the way we go about realizing them is frequently "off" somehow. The stark fact is that too many large SOA initiatives often crash and burn (an astonishing 80% of them still fail to meet expectations or fail completely.) At the same time, most IT departments are simultaneously struggling with a constellation of lower level yet closely related concerns that often lack strategic context without SOA. These include APM, data center modernization, virtualization, workflow management, and IT governance to name just a few.
What's become clear is that the different human perspectives that each and every stakeholder within IT and the business brings to the table forms a constellation of competing concerns and goals which poses serious obstacles to realizing serious enterprise-wide IT efforts of any kind. This conflict appears particularly true of SOA because it cuts such a cross-boundary swath across an organization. Tribal parochialism can be a real threat to the ongoing health of a business as it attempts to evolve to address the realities of the 21st century business climate. It also makes SOA governance ultimately unlikely to make much headway, putting initial progress with SOA in long-term doubt.
I've been having some intriguing public discussions on this topic with Michael Krigsman, Dana Gardner, Miko Matsumura, and many others over the last couple of months trying to understand this current state of affairs. The goal: To map out how SOA governance might be more fully reconceived to meet these challenges.
What does next-generation SOA governance look like?
A more sophisticated and enlightened view of SOA governance today might divide the practice into three major axis. The first is the people aspect, where issues such as politics, corporate culture, risk tolerance, and other criteria will impact the adoption and evolution of SOA. The second is the structured time dimension of SOA, or process, whereby governance mechanisms carry themselves out. These include run-time capabilities such as automated compliance monitoring and auditing as well as more strategic, non-technical activities such as SOA certification processes, architecture reviews, and non-compliance resolution procedures.
The last is technology itself, the means through which much of the advantage of SOA is actually delivered. These include the full gamut of artifacts to be governed ranging from the services themselves, applications connected to them or built on top of them (composite apps or mashups), as well as the surrounding fabric of the network and its attendant SLAs and underlying business data.
As we can see from a map of the most common approaches to SOA governance below, there is an extensive tool focus today that emphasizes both design-time and run-time governance. These are important and necessary capabilities, but invariably the real issues with SOA are high "up the stack" and reside in what are often called "tribal concerns" in the people axis.
That's not to say that there aren't a full range of options today that cover a wide spectrum of SOA Governance concerns, from strategic services from major vendors like Accenture, HP, and IBM to well-documented governance frameworks and methodologies such as TOGAF 9, Val IT, and ISO 20000.
In total, these products, tools, and methods provide a fairly complete set of resources for those that are looking to govern their SOA efforts enterprise-wide. But the map below also highlights the weakness in one of the three axes: people. There exist many offerings from a technical and process standpoint, but the ones that focus on the tribal issues are either major commercial consulting services or fairly generic IT architecture and service management process standards.
What often seems to be lacking is real engagement with the organization. As Michael Krigsman observed recently in our SOA governance discussions with the industry in Modern SOA governance: Adoption and measurement:
Governance is essential because a centralized, big brother attitude toward technology deployment rapidly becomes impractical in the SOA world. Instead, success relies on allowing decentralized service development within the confines of agreed-upon rules. This rapidly becomes a social and organizational problem that goes beyond technology alone.
Well-known SOA expert Miko Matsumura, in a recent joint podcast, concurred:
As you start to enforce governance policies and rules, how does the community respond and how do you dynamically monitor their behavior, because this transitional period is the one of greatest risk. I call this the "danger zone" in my book, SOA Adoption for Dummies....
SOA adoption is dynamic. The computer parts are frankly deterministic but behavior is situational and sensitive.
This goes to the issue of how you connect governance to the internal community which you want to both embrace SOA principles and services and satisfy their own needs and objectives. I've been examining examples of extremely successful public SOAs over the last couple of years and I've frequently noted that it's when SOA is treated as a business in its own right, it tends to bring into the balance the forces enabling good governance with SOA adoption that actively drives forward better business outcomes.
But often the real problem of SOA adoption and governance is not having a connection to the broader internal business community at all (again, very unlike the successful open APIs of the Web). And this is where Web 2.0 and Enterprise 2.0 approaches to collective intelligence can come in, which Krigsman calls a "killer app" for SOA. In an upcoming post in early February I will explore the emerging details we're learning in how to connect an SOA effort to the community to enable much more effective adoption and governance. After all, as Miko told me recently, in the end "SOA adoption is social." And unless the social aspects of SOA are addressed head-on, the most important axis of adoption and governance will under perform, often significantly.
Sidenote: The rapid emergence of cloud computing is also hampering design-time governance efforts as Joe McKendrick and David Linthicum explored recently, bringing back technical issues with SOA governance that had once been largely resolved.
Disclaimer: I have been partners with or had client relationships with some of the companies mentioned in this post including Accenture, Microsoft, IBM, Progress, Asuret, and Alcatel-Lucent.
What do you see as the biggest SOA adoption and governance challenges in your organization? Please leave your feedback in comments below.