Recently in Enterprise Architecture Category

Continuing the discussion from last week about how we might look to related disciplines and industries to improve our ability to execute on SOA, I had an interesting discussion last week with my good friend Stefan Andreason, CTO of mashup data provider Kapow, and the CEO of StrikeIron, Bob Brauer.

Both companies have a long track record in helping organizations access their data via services and they were discussing with me their new joint effort to combine services in the "head" of the services market (StrikeIron) with the tail of the services market (Kapow) and create a consistent delivery and management capability around them. I think it's a smart move that reflects a growing realization about the very nature of enterprise services themselves.

Recognizing the Service Diversity in the Real World

The premise is that businesses and the Web have a lot more data than can be reached through traditional means, either as services or otherwise. Though we have been service-enabling our IT systems and data silos for a decade, there's still so much work to be done that we have to almost radically step up our our ability to deliver on them.

One of the big elephants in the room these days is that the Web has risen in the last few years as one of the most important content "silos" to many firms. This has led to the growing understanding that we need fundamentally more effective ways to scale up both the creation and consumption of services. As part of this, it's also started to become clear that in the real world there is a broader continuum of types of services that organizations should support in their SOA efforts. Despite positive moves in certain directions -- I see many firms adding JSON to the mix for example -- we've been largely ignoring a broad class of network delivery models that should probably be first class citizens in a modern SOA.

The Services Continuum: SOA, SOAP, REST, open Web APIs, and Semantic Web, Linked Data

A fuller services continuum -- you can click to enlarge the visual above -- starts with the traditional Web services that we use in our SOA efforts today and can be as simple as WS-I Basic Profile (SOAP + WSDL) or more sophisticated versions that include support for security, transactions, policy, and more (often referred to in shorthand as WS-*). The continuum ends with the lightweight services of the Web such as REST, RSS, and ATOM. It even includes Web content itself, something that we're learning can be treated as services since everything on the Web or local intranet can be accessed in the same way, via HTTP, which also happens to be the foundational protocol of Web services in general.

This highlights a certain level of myopia we still have in enterprise architecture, where we tend to look at services with a very narrow definition, when we are likely much better served by a more encompassing view. While we might have at most a few hundred well-defined services in a typical organization today, there are literally millions of them on the open Web and many more inside our organizations under this larger umbrella. Whether we are trying to integrate systems more quickly, discover and get access to information more effectively, or rapidly building business apps based on these services (mashups), this more expansive and heterogeneous notion of services gives us a much bigger and richer pallette to work with to meet our needs.

5 Ways To Expand Your Services Continuum

To expand our conception then of modern SOA and enterprise services, we must consider the following:

  1. Make it formally acceptable to use a broader range of services in the organization. Not every job requires a hammer as the tool and the same goes for the ypes of services we use. For example, transaction processing systems are very different than say, analytical systems, where the former needs more structure and control and the latter might need less. The architecture group, CTO, or other governing bodies can enable the business to get access to far more value by green lighting more types of services to use, particularly feeds, Web APIs, and Internet/intranet content-as-a-service.
  2. Establishing guidance across the services continuum. Clearly define what architects, developers, designers, and end-users can do with this broader range of services -- both in the enterprise and out on the Web -- and under what conditions.
  3. Adopt management, administration, and governance capabilities that support a fuller services continuum. Right now, services of all kinds are already being consumed in many organizations but by bringing the full continuum into the formal IT fold, the quality and predictability issues can often be made to go away and business opportunities increased.
  4. Ensure internal customers have the ability to consume new type of services. This actually goes for both traditional enterprise services as well as the full continuum itself; focusing on driving consumption hasn't been a high priority in many organizations and both technical and line of business workers are too often lacking the ability readily incorporate services in their work, even when they are available. Updating local frameworks, acquiring mashup tools, and using service registries that can manage the full spectrum of modern services are examples of how to address this. For example, StrikeIron even has an interesting method of connecting Excel to this full services landscape so that all business users can consume them. This is a big change in thinking about increasing service consumption and opens up intriguing possibilities.
  5. Create many more services to consume. It's a supply and demand problem, in that if there are not enough services, they will be very little demand, not too much. Other ways will be found to solve the problem of accessing data or interacting with trading partners. Solutions such as Kapow and WSO2 Data Services server are good examples of how to create a much bigger services landscape in short amounts of time. I'm not advocating a speculative "field of dreams" approach here, but the ability to find, build, or provision services right when they're needed.

What are you seeing with data services? Are you seeing the broadening of the idea of enterprise and Web services?

I've been spending the year assessing where enterprise architecture -- and SOA in particular -- are headed as we begin to leave the first decade of the 21st century. The good news: It's clear that EA and SOA are delivering real value to most businesses today. However as we'll see, there is also considerable room for performance improvement. Trying to map out the direction of these disciplines has resulted in quite a snapshot with many interesting details and nuances that I'll share here throughout the rest of the year.

More immediately of interest we can see some trends beginning to stand out clearly right now:

1) Modern software architecture, with SOA as the top-level organizing principle, seems to have more and more trouble keeping up with the rate of change in most organizations. Not only have the timelines and schedules of IT and architecture groups never aligned well with business activities, it's now becoming an endemic problem as I've shared experiences with architects around the world recently. There's a growing sense that the business is "pulling away" and despite enterprise architecture and SOA being intentionally proactive, the default stance is one of short-term reactive response and trying to "clean up afterward", an activity which few businesses want to pay for. In short, our traditional services delivery models seem to move too slowly these days to embrace business opportunities as they occur.

2. Consumption of SOA and service-based IT is still too low. Depressingly low in some cases. The early "field of dreams" approach to services that some organizations practiced (build them and they will come) combined with the proliferation of JBOWS (just a bunch of Web services) finally morphed at last in many organizations into more mature services landscapes with some actual governance. But now we see that the subsequent complexity, rarefied skills and tools, and additional constraints ended up stamping out a lot of SOA consumption at the edge. These days the evidence is beginning to mount (presented below) that a certain style of services -- combined with mechanisms that actively drive consumption and outfitted with real business models -- could address this. There are other emerging solutions to drive consumption of SOA as well. Without consumption and uptake, SOA cannot access the "return" in ROI. Systematic removal of the obstacles to participation in SOA is the goal in this view.


3. The focus on SOA still tends to be on the [over]engineering of seams and processes instead of removing constraints on the business and increasing ready access to value. We are still not providing good enough reach to our enterprise data, IT systems, or people. Despite heavy investments in IT for 30 years and the openness and interoperability intents of SOA, the majority of our enterprise data remains submerged and inaccessible to most business users, silos are still prevalent, and there still lacks the human dimension, the latter which can represent the use of more effective architectures of participation or the involvement of more than IT in the strategic development of SOA.

While important cross-cutting concerns -- like security efforts or the imposed siloed architectures of enterprise software suites (ERP, CRM, HRM, etc) or even turf boundaries (see Conway's Law) -- are also responsible for many of these issues, at the core seems to be a project-oriented focus instead of a product-oriented focus with SOA. This is not to say that good engineering is not important, but that we are too often aiming at the wrong part of the SOA tolerance continuum.

Looking to successful models of SOA for answers

As an industry it's also apparent that we have too few examples of good SOA to point to and use as reference models. Most success stories are behind the walls of other organizations that are just as inclined to keep the secrets of success to themselves. Best practices and effective approaches therefore are not always likely to spread when they should. In fact, it's fairly easy to argue that the evolution of SOA has often moved far faster in standards bodies than by validation of effective new ideas in the field that are only then adopted and standardized when they work well.

Interestingly, this means the openness and visibility of today's modern Web seems to provide living examples of what's effective at driving forward business results with SOA. By this I mean when it addresses the key issues around improving the response to opportunities and directly enabling self-service, all while remaining highly scalable and protecting users and data. While I've been writing about Web-Oriented Architecture (WOA) for some time now, it's in truth a parallel "track" for SOA that's evolved organically in the wilds of the online world to meet many of the same challenges that we have in our organizations today.

Related: Fixing Enterprise Architecture: Balancing the Forces of Change in the Modern Organization

Key to the story of WOA is that at least one critical feedback loop is present in online businesses that is much less evident in most SOA efforts today: Their business will fundamentally thrive or die based on the adoption of their services. Most new online businesses offer an API (the Web version of SOA) at the very outset these days and it's now common for the services themselves -- instead of the visual Web pages -- to be the dominant point of usage early on. Online firms offer APIs in easy to use formats and then market and evangelize them extensively precisely because there is an urgent need and these are the best ways to drive adoption. There is also more direct and immediate business value in this model since carefully measured use and consumption is the yardstick by which online firms are measured. It's also what they monetize, an entire area that's seriously underdeveloped in enterprise SOA for both internal and external customers. This is a yardstick that would change a lot of what we focus on in SOA.

Imagine how we might behave and prioritize our efforts if our success was largely based on the number of points and overall level of consumption of SOA services, internally as well as partners and customers. Unfortunately, as Joe McKendrick noted yesterday in "SOAs too big to fail", we may be increasingly isolated from that pressure as the business becomes more and more dependent on SOA. Ironically, this seems to be encouraging an organizational structure (which often is the IT department itself) that is isolated from market pressures.

How WOA can help SOA reach the next level of performance

While there are certainly many aspects of WOA that need to have an enterprise context added to them to work for traditional businesses, you can see from the visual above that a successful and capable service-based approach has emerged naturally side-by-side with classical SOA. At their roots, they both have great commonalities (HTTP, interoperability, and recombinant software), and some key differences as well. Intriguingly however, these often line up with many of the issues we're now seeing holding back the next level in business performance of SOA:

  • Responding to change faster. Reducing execution chokepoints by making SOA more self-service (running your SOA like a business) by broader audiences in the organization.
  • Increasing consumption. Using new service delivery models that make SOA the easiest, cheapest, and fastest way to solve problems.
  • Actively enabling access to value. Opening up broad access to data and people using models such as deeply linked REST-based data webs and open supply chains. Adopt browser-based approaches to consumption such as enterprise mashups tools and user distributable widgets that project the SOA across the organization.

We certainly don't need to throw away everything that we've done with SOA to achieve these performance improvements, far from it. However, we must change much of the focus on what we do and how we do it. There are also some key additions to technical approaches that will be required such as adopting enterprise REST, making IT systems more mashup friendly, and updating existing delivery models such as portals and intranets to be more malleable, self-service, and connected to SOA data and resources.

The bottom line: Since many of the very best examples of SOA that works are the models we seeing being proven out in large scale on the Web, we must learn as much as we can from them (and the price is right, even the most compelling lessons here are free.) While enterprise SOA will never be identical to Internet-based WOA, we can borrow the best ideas from the success stories that have emerged on the Global SOA (which is what I call the service-enabled side of the Internet).

We are also starting to see evidence that a more mature melding of Web-oriented service models with the enterprise is actually starting to happen. For example, an InformationWeek survey this year showed that on the technical side of the SOA/WOA stack, SOAP adoption appears to be dropping in enterprises (54% a year ago to a projected 42% in the next 18 months) while REST is on the rise (14% to 24% over the same time frame). The recent announcement of the opening up of EMML is also showing that WOA concepts around lightweight composite applications (mashups) have moved into SOA territory.

While emerging initiatives such as JBOSS's REST-* may or may not help with this transition, as more of the foundation of WOA is available in organizations, richer and more effective outcomes on the business side of the the WOA equation are possible as well. No doubt this will take years, but this evidence and others shows that we are already beginning to incorporate the lessons of WOA into enterprise SOA.

In an upcoming post, I'll look at emerging new SOA practices that are coming from other sources as well.

Note: I'll be holding a discussion on The Future of SOA and enterprise REST at ebizQ's SOA in Action on October 28th. Please register to participate today.

Are Web-based approaches providing insight to your SOA effort? Or are you getting effective new ideas from elsewhere?