The common approach to SOA is to define sets of services that may exist physically within the data center, on a public cloud, or perhaps in both places (hybrid cloud). The degree that these services provide functional behavior, and functional behavior bound to data, really goes to the value. However, I would also say that we need to consider the quality of the services design, the granularity, or how well the services approach a "functional primitive," and thus providing more value.
The ability to design, define, and develop services at a primitive level goes to the value of those services considering that it's much easier to create composites out of groups of primitive services. Thus, allowing you to mix and match services to live up to the exact purpose/requirements of the composite service, process, or application, rather than attempting to deal with services that are to high level, or course grained, and thus not an exact fit for your requirements.
An analogy that I would use is that it's much easier to build a custom designed home using wood, fasteners, or other materials than to build a house from pieces of another home. The more primitive the parts, the more flexibility you have when building solutions. Rather than trying to reuse a living room.
This is of particular importance when considering cloud computing. You want to expose services that provide very primitive and low-level functions, and thus are more useful to those that want to consume them. Public clouds providers get this, and thus provide many primitive APIs that seem complex, but provide much more value to those looking to create business solution.
It's important that you consider the levels that you define services, and the more primitive the better. I understand that is a bit counterintuitive, but many aspects of good SOA are. When thinking SOA, tell everyone you're taking the most primitive approach.
Okay, you need to push your customer data to Salesforce.com, and back again? There are dozens of technologies, cloud and not cloud, which can make this happen. Moreover, there are many best practices and perhaps pre-built templates that are able to make this quick and easy.
But, what if you're not using Salesfoce.com, and your cloud is a rather complex IaaS or PaaS cloud, that is not as popular and thus not as well supported with templates and best practices? Now what?
Well, you're back in the days when integration was uncharted territory and when you had to be a bit creative when attempting to exchange information with one complex and abstract system with another. This means mapping data, transformation and routing logic, adapters, many of the old school integration concepts seem to be a lost art these days. Just because your source or target system is a cloud and not a traditional system, that does not make it any easier.
The good news is that there are an awful lot of effective integration technologies around these days, most of them on-premise with a few of them cloud-delivered. But, learning to use these products still requires that you have a project mentality when approaching cloud-to-enterprise integration, and it's not an afterthought, as it is many times. This means time, money, and learning that many enterprises have not dialed into their cloud enablement projects.
Many smaller consulting firms are benefiting form this confusion and are out there promoting their ability to connect stuff in your data center with stuff in the cloud. Most fall way short in delivering the value and promise of cloud integration, and I'm seeing far too many primitive connections, such as custom programmed interfaces and FTP solutions out there. That's a dumb option these days, considering that the problem as already been solved by others.
I suspect that integration will continue to be an undervalued part of cloud computing, until it becomes the cause of many cloud computing project failures. Time to stop underestimating that work that needs to be done here.
As presented in Network World, "The Equal Employment Opportunity Commission (EEOC) expects to save 40% over the next five years by switching its financial management application to a cloud computing vendor -- a sign of the massive savings to come from the U.S. federal government's shift to the software-as-a-service model." Good, but not great.
The reality is that US government has a lot of IT fat that can be cut through the use of cloud computing, with leveraging SaaS-based applications being the low hanging fruit. What's great about SaaS is that that the business case is obvious, and the savings are typically between 40 and 60 percent. However, what's not so great about SaaS is that you're only dealing with a single application domain and not the architecture holistically.
While the savings that the US government, and we as taxpayers, can enjoy from cloud computing is significant perhaps tactical moves such as leveraging a single SaaS application only masks larger more systemic issues. Indeed, what's truly needed is an overall strategy around the use of cloud, and the architectural steps to get there. This includes the use of other cloud solutions, such as IaaS and PaaS, as well as SaaS.
The problem is that architectural change around the use of new technology, such as cloud computing, is hard while just migrating from a single on premise application to a SaaS app is easy and quick. However, the former offers many more efficiencies and cost savings when considering both the economies of the technologies as well as the agility their use brings.
So, when government agencies think cloud they need to thing long term and strategic, and not short term and tactical. We will all be much happier with the end result.
]]>Loraine Lawson wrote a compelling blog post: "Did SOA Deliver on Integration Promises?" Great question.
"So did SOA solve integration? No. But then again, no one ever promised you that. As Neil observes, we'll probably never see a 'turnkey enterprise integration solution,' but that's probably a good thing - after all, organizations have different needs, and such a solution would require an Orwellian-level of standardization."
The fact of the matter is that SOA and integration are two different, but interrelated concepts. SOA is a way of doing architecture, where integration may be a result of that architecture. SOA does not set out to do integration, but it maybe a byproduct of SOA. Confused yet?
Truth-be-told integration is a deliberate approach, not a byproduct. Thus, you need to have an integration strategy and architecture that's a part of your SOA, and not just a desired outcome. You'll never get there, trust me.
The issue is that there are two architectural patterns at play here.
First, is the objective to externalize both behavior and data as sets of services that can be configured and reconfigured into solutions. That's at the heart of SOA, and the integration typically occurs within the composite applications and processes that are created from the services.
Second, is the objective to replicate information from source to target systems, making sure that information is shared between disparate applications or complete systems, and that the semantics are managed. This is the objective of integration, and was at the heart of the architectural pattern of EAI.
Clearly, integration is a deliberate action and thus has to be dealt with within architecture, including SOA. Thus, SOA won't solve your integration problems; you have to address those directly.
]]>However, many are not chasing cloud computing the right way, missing many of the architectural advantages. Instead they are just tossing things out of the enterprise onto private and public clouds and hoping for the best. Making things worse, many in larger enterprise clients are not seeing the forest through the trees, or in this case the architecture through the clouds. So, you have both parties taking a reactive versus a proactive approach to the cloud.
Missing is good architectural context supporting the use of cloud computing. Or, the ability to create an overall strategic plan and architectural framework, and then looking at how cloud computing fits into this framework now, and into the future. Typically that means leveraging SOA approaches and patterns.
That message seems to fall on death ears these days, and most disturbingly those death ears seem to be attached to consulting organizations that have the trust of enterprises to take their IT to the next level. The end result will be failed cloud computing projects, with the blame being put on the technology. It's really the lack of strategic planning and architecture that's at the heart of the problem.
The concept of SOA, as related to cloud computing is simple. You need to understand the existing and future state architecture before you begin selecting platforms and technology, including cloud computing. Once you have that understanding its relatively easy to figure out where SaaS, IaaS, and PaaS come into play, or not. Moreover, creating a roadmap for implementation and migration over time, typically a 3 to 5 year horizon.
We need to get good at this quick, else cloud computing will do little good.
The issue is one of context, more so than technology when considering that we've been doing integration well for over 15 years, and thus it's just a matter of carrying those concepts and technology to the world of the cloud. However, as it was back in the 90s when I wrote the EAI book, many are approaching cloud integration as if we are starting over. That's a big mistake.
What's key to remember about integration, cloud or not, is that it's actually about the free flow of information between systems that deal with information differently. Thus, you have to adjust information as it's transmitted between systems, or how each deals with system semantics. For example, in moving information from SAP ERP to Salesforce.com, at some time you need to deal with the different ways that each structures data around the concepts of customer, sales, inventory, etc..
Not a ton changes when dealing with cloud computing, other than the fact that you may be dealing with systems that are outside of the firewall, and not under you're direct control. However, they also typically provide well-defined and easy to use interfaces, or APIs, which allow access to core information or services. Indeed, I would consider it much easier to connect and integrate existing SaaS and IaaS clouds than traditional enterprise systems.
The core message here is that we need to learn from the past, and don't assume we're starting from scratch when dealing with cloud computing. The patterns, the technology, the problems, and the solutions are largely the same.
First, we don't own nor control the cloud computing-based systems, thus we have to deal with what they provide us, including the limitations, and typically can't change it. Thus, we can't do some types of testing such as finding the saturation points of the cloud computing platform to determine the upward limitations on scaling, or attempt to determine how to crash the cloud computing system. That type of testing may get you a nasty e-mail. Or, white box testing the underlying platform or services, meaning viewing the code, is also, not supported by most cloud computing providers, but clearly something you can do if you own and control the systems under test.
Second, the patterns of usage are going to be different, including how one system interacts with another, from enterprise to cloud. Traditionally, we test systems that are on-premise, and almost never test a system that we cannot see nor touch. This includes issues with Internet connectivity.
Third, we are testing systems that are contractually obligated to provide computing service to our architecture, and thus we need a way to validate that those services are being provided now, and into the future. Thus, testing takes on a legal aspect, since if you find that the service is not being delivered in the manner outlined in the contract, you can take action.
Finally, cloud computing is relatively new. As such, IT is a bit suspicious about the lack of control. Rigorous and well-defined testing, will eliminate many of those fears. We must be hyper diligent to reduce the chances of failure, and work around the fear of what's new.
For our purposes, we can call policies that are more general in nature macro policies, and policies that are specific to a particular service as micro policies.
Macro Policies
Macro policies are those policies that IT leaders typically create, such as the enterprise architect, to address larger sweeping issues that cover many services, the data, the processes, and the applications. Examples of macro policies include:
• All metadata must adhere to an approved semantic model, on-premise and cloud computing-based.
• All services must return a response in .05 seconds for on-premise and .10 for cloud computing-based.
• Changes to processes have to be approved by a business leader.
• All services must be built using Java.
The idea is that we have some general rules that control how the system is developed, redeveloped, and monitored. Thus, macro polices do indeed exist as established simple rules, such as the ones listed above, or set processes that must be followed. For example, there could be a process to address how the database is changed, including 20 steps that must be followed, from initiation of the change to acceptance testing. Another example is the process of registering a new user on the cloud computing platform. Or, any process that reduces operational risks.
Many have a tendency to roll their eyes at these kinds of controls that are placed around automation. I'm sure you have many that exist within your IT shop now. They may also push back on extending these governance concepts to cloud computing. However, the core value of implementing macro policies is to reduce risk and save money.
The trick is to strike a balance between too many macro policies that hurt productivity, or too few that raise the chance that something bad will happen. Not an easy thing, but a good rule of thumb is that your IT department should spend approximately 5 percent of their time dealing with issues around macro polices. If you spend more time than that, perhaps you're over-governing. Less than that, or if you have disaster after disaster happen, perhaps you can put in more macro policies to place more processes around the management of IT resources, on-premise or cloud computing-based.
Micro Policies
Micro or service-based polices typically deal with a policy instance around a particular service, process, or data element. They are related to macro policies in that macro policies define what needs to be done, whereas the micro policies define how a policy is carried out at the lowest level of granularity.
Examples of micro policies include:
• Only those from HR can leverage Get_Sal_Info services.
• No more than 1 application, service, or process at a time can access the Update_Customer_Data service.
• The Sales_Amount data element can only be updated by the DBA, and not the developers.
• The response time from the get_customer_credit service must be less than .0001 seconds.
Micro policies are very specific, and typically destined for implementation within service governance technology that can track and implement these types of policies.
Standards are a double edged sword; they clearly provide some value by protecting you from vendor-specific standards, in this case, cloud lock-in. However, they can delay things as enterprise ITs wait for the standards to emerge. Moreover, they may not live up to expectations when they do arrive, and not provide the anticipated value.
Standards should be driven by existing technologies, rather than by trying to define new standards approaches for new technologies. While the latter does occasionally work, more often it leads to design-by-committee and poor technology. Past failures around standards should make this less of an issue in the world of cloud computing.
So, when considering SOA and cloud computing standards, take a few things into consideration:
• Standards should be driven by three or more technology vendors that actually plan to employ the standard. Watch out for standards that include just one vendor and many consulting organizations. Or, are just driven by marketing.
• Standards should be well-defined. This means the devil is in the details, and a true standard should be defined in detail all the way down to the code level. Conceptual standards that are nothing but white papers are worthless.
• Standards should be in wide use. This means that many projects leverage this standard and the technology that uses the standard, and they are successful with both. In many instances you'll find that standards are still concepts, and not yet leveraged by technology consumers.
• Standards should be driven by the end users, not the vendors. At least, that's the way it should be in a perfect world. While the vendors may have had a hand in creating the standards, the consumers of the technology should be the ones driving the definition and direction. Standards that are defined and maintained by vendors often fail to capture the hearts and minds, while standards maintained by technology consumers typically provide more value for the end user and thus live a longer life.
The lack of an architecture -- typically, the lack of a SOA -- is a recipe for failure in the world of cloud computing. An architecture provides the structure necessary to mesh your existing enterprise IT assets with the emerging world of cloud computing. Most who leverage clouds, PaaS, IaaS, or SaaS, understand the dilemma and quickly turn to basic architecture and planning...only to find that those 'in the know' are nowhere to be found.
Good SOA architects are a rare species. Many who claim to have mad SOA skills come up short. The trend is to leverage whatever the next magical and hyped technology is in the hopes that no one will notice that the existing architecture is a huge mess, and the addition of cloud computing resources will just make it messier.
Making matters worse are the numbers of SOA technology vendors who have falsely position their technology as "cloud computing technology," when they should be focused on SOA leading to successful cloud computing. There is a huge difference. This vendor hype has just added to confusion around both the concepts of cloud computing and SOA, and the end users are once again looking to toss technology at problems that really need better architectural thinking.
Clearly, we don't have enough SOA A list players to go around as cloud computing explodes.
Been thinking. Considering that loose coupling is a foundation of SOA, and I would say cloud computing as well, perhaps it's a good idea to break down loose coupling into a few basic patterns: Location independence, communication independence, security independence, and instance independence.
Location independence refers to the notion that it matters not where the service exists, the other components that need to leverage the service can discover it within a directory and leverage it through the late binding process. This comes in handy when you're leveraging services that are consistently changing physical and logical locations, especially services outside of your organization that you may not own such as cloud-delivered resources. Your risk calculation service may exist on premise on Monday and within the cloud on Tuesday, and it should make no difference to you.
Dynamic discovery is key to this concept, meaning that calling components can locate service information as needed, and without having to bind tightly to the service. Typically these services are private, shared, or public services as they exist within the directory.
Communications independence means that all components can talk to each other no matter how they communicate at the interface or protocol levels. Thus, we leverage enabling standards, such as Web services, to mediate the protocol and interface difference.
Security independence refers to the concept of mediating the difference between security models in and between components. This is a bit difficult to pull off, but necessary to any SOA. To enable this pattern, you have to leverage a federated security system that's able to create trust between components, no matter what security model is local to the components. This has been the primary force behind the number of federated security standards that have emerged in support of a loosely coupled model and web services.
Instance independence means that the architecture should support component-to-component communications using both a synchronous and asynchronous model, and not require that the other component be in any particular state before receiving the request, or, the message. Thus, if done right, all of the services should be able to service any requesting component, asynchronously, as well as retain and manage state no matter what the sequencing is.
The need for loosely coupled architecture within your cloud computing solution is really not the question. If you leverage cloud computing correctly, other than in some rare circumstances, you should have a loosely coupled architecture. However, analysis and planning are also part of the mix...understanding your requirements and how each component of your architecture should leverage the other components of your architecture. Leverage the coupling model that works for you.
SOA governance is a concept used for activities related to exercising control over services in an SOA, including tracking the services, monitoring the service, and controlling changes made to the services, simple put. The trouble comes in when SOA governance vendors attempt to define SOA governance around their technology, all with different approaches to SOA governance. Thus, it's important that those building SOAs within the enterprise take a step back and understand what really need to support the concept of SOA governance.
The value of SOA governance is pretty simple. Since services make up the foundation of an SOA, and are at their essence the behavior and information from existing systems externalized, it's critical to make sure that those accessing, creating, and changing services do so using a well controlled and orderly mechanism. Those of you, who already have governance in place, typically around enterprise architecture efforts, will be happy to know that SOA governance does not replace those processes, but becomes a mechanism within the larger enterprise governance concept.
People and processes are first thing on the list to get under control before you begin to toss technology at this problem. This means establishing an understanding of SOA governance within the team members, including why it's important, who's involved, and the core processes that are to be follow to make SOA governance work. Indeed, when creating the core SOA governance strategy should really be independent of the technology. The technology will change over the years, but the core processes and discipline should be relatively durable over time.
With the New Year right around the corner, and most of the 2011 prediction blogs already posted, perhaps it's time to look at the true trends that will occur in the world of SOA and Cloud Computing in 2011. I'll be brief so you can get back to your eggnog.
Trend 1: Cloud providers become more aware of SOA as an architectural approach to enable cloud.
Most cloud provides consider SOA like Big Foot. They know it's out there, they see glimpses of it once in a while, but they have yet to capture it and put it in a Zoo. However, as 2011 progresses that will certainly change, and I'm seeing more cloud providers become interested in the value of SOA in the application of their cloud services within enterprises, no matter if it's PaaS, SaaS, or IaaS.
Trend 2: SOA governance technology begins to provide true value.
There is certainly a tipping point where the number of services gets to such a level that you need service governance technology to manage it for you. That tipping point was reached in 2010 for many enterprises, and it will only get worse in 2011. The use of a good service governance technology platform is absolutely essential for a successful cloud meets the enterprise deployment.
Trend 3: Centralized sharing of services becomes a focus.
While we've been learning to leverage services that come out of the clouds, and services that we externalize into the cloud, however there has not been a lot of focus on how services are hosted and discovered for sharing among enterprises. I suspect that clouds will become better at allowing enterprises to onboard services, and re-share those services in 2011. Kind of like eBay for application and infrastructure services.
The trend is your friend. Good luck in 2011.
Strategic decisions about cloud computing should both draw upon and inform the EA. An organization must have a mature and well formed understanding of its architecture components (e.g., business processes, services, applications and data) to make meaningful decisions related to cloud computing, such as whether a move to the cloud is advantageous, what services most lend themselves to a cloud deployment, and what cloud deployment model (e.g., private, public) makes the most sense.
There are three key roles for EA in facilitating cloud computing strategy and planning:
Front-end decision support. An organization's EA should inform decisions about the desirability of a move to a cloud environment, what services should move to the cloud, and the appropriate deployment models. Existing business processes, services and resources should be analyzed through the lens of cloud characteristics and quality dimensions, such as elasticity, reliability, and security.
Cloud implementation planning support. During the period of transition, an EA provides the current inventory of services and the roadmap to which services are being deployed into what clouds. As new business needs emerge, the EA provides solution architects with a view of what has been deployed in the cloud already and what is planned for future deployment.
Enterprise context. Sharing and reuse of services and resources long have been primary objectives of EA. In the context of a cloud strategy, EA provides a critical enterprise view to ensure cloud decisions are optimized at the enterprise level and independent decisions on cloud-based point solutions are viewed in a broader context.
In the world of cloud computing where storage and compute are accessed using well-defined APIs, we also have available to us information that can be as easily accessed, also using well-defined APIs. Typically, these providers offer up zip code or address validation and lookup, payment processing, or other services that validate or complete data. In other words, information-as-a-service.
One of the players out there is Postcode Anywhere, who provide Web services which are accessible over the Internet providing these data validation services.
The idea is simple. You link to the Web service to perform the validation service typically using an URL. You pass in the parameters required, such as a zip code, and it passes back the result.
The advantage of leveraging this kind of service is that you don't have to maintain the data yourself, which is a daunting if not impossible task. Moreover, you can mix and match these services within applications or processes, as needed. Also, considering that the interfaces are standardized, you don't need to relearn them as you move from API to API.
Applications for these types of services are many, including address validation and payment processing for a commerce site, or perhaps even data cleansing operations in larger batches. Also, you're able to embed these services within other clouds, such as IaaS as needed.
These micro-clouds are not at all new, and they have been refining their services over many years. Now that it's more acceptable to leverage cloud computing, you need to make sure that you look at information-as-a-service as part of your cloud computing and SOA strategy.
]]>