Evaluating Investments in Systematic Application Integration: Configuring Your Enterprise Framework To Improve Your Bottom Line
By Roy Schulte, Vice President and Distinguished Analyst, Gartner
Almost any management initiative that leverages IT to support a new or enhanced business process will require some attention to the issue of application integration. Integration is the challenge of getting independently-developed application systems to work together, i.e., sharing data and achieving cooperation among various applications that may support different business units and run in different data centers or even in separate enterprises.
Integration was traditionally implemented as part of the application development project, using point-to-point links and conventional development languages and software tools. During the past seven years, however, a newer systematic approach to application integration has emerged, exploiting one or more of the following:
integration suites -- a type of system software that provides transformation and intelligent connectivity services, along with other related functions;
business process managers (BPM) -- a software engine tracks and controls the flow of composite applications or multistep business processes;
prebuilt adapters for connecting application systems into the integration infrastructure;
adapter development tools;
packaged integrating processes (PIPs) and packaged composite applications (PCAs) which incorporate prebuilt ("off the shelf") process flows and transformations for an end-to-end integration solution;
a central integration competency center -- a relatively new part of the IS organization that coordinates the development and operational activities associated with integration across multiple application projects and business units
This report provides a general framework for understanding the costs and benefits of systematic application integration that uses one or more of these feturees. We contrast systematic integration using purpose-built high level integration middleware tools to implementing the same set of applications and integration processes using traditional approaches to development.
Summary: The aspects of systematic integration listed above are independent decisions with separate benefits, i.e., an enterprise can choose to implement virtually any combination of them:
Broker payback depends mostly on the scope of the interface complexity, i.e., more complexity in transformation and routing logic indicates the opportunity for greater payback by using a broker.
BPM payback depends mostly on the nature of the business processes and the associated changes that are made outside of the IS department. Complex (with many steps, especially if executed by disparate business units), dynamic (fast changing) processes will achieve greater payback than simple stable processes.
A well-run integration competency center will have a positive payback all but the smallest scenarios, regardless of whether any commercial broker or BPM tools are used.
Systematic integration requires incremental spending on the new types of software, and substantial time and effort by the IS technical staff for training and technical chores. It can sometimes provide "hard savings" within the IS department in the form of reduced time and effort for developing and maintaining the application systems and their network of interactions. However, the more commonly achieved and more important benefits of systematic integration appear outside of the IS department in the form of:
enhanced enterprise agility (i.e., the ability to process work faster, and the flexibility to modify those processes more quickly);
the ability to manage and disseminate knowledge among applications and business units better (data quality in a sense);
improved quality in a range of manufacturing, sales, and administrative processes;
and reduced cost of labor for non-IS clerical and management personnel.
Systematic Integration has the following implications on project time tables:
Depending on the nature of the specific applications, systematic integration can shorten the initial development time for large, complex new projects involving integration, and it is often more helpful in cutting the time required to implement subsequent additions or modifications to an existing set of interconnected, integrated applications.
Systematic integration is a "break even" proposition in terms of time-to-completion for new integration projects of moderate size and complexity (traditional integration can be implemented about as quickly).
Small projects will take longer to complete if done systematically. Traditional approaches to integration will be faster, especially for the initial implementation.
Note that application integration, in a general sense, is an inherent part of modern application development. The only way to avoid spending any money on application integration is to eliminate new development or enhancement projects altogether. So the real question about application integration is whether this inescapable chore of integration should be done as part of the application development project using traditional development practices and software tools; or if a new-style, centrally coordinated, systematic approach to application integration (generally involving modern middleware such as an integration suite and/or business process management software) is cost-justified.
Integration typically consumes about 35 percent of the cost of a traditional application system. This includes the cost of designing and coding program changes; buying development tools, utilities, database gateways or other middleware; and the hardware, environmental, and operational overhead to run this integration logic and maintain related databases. However, most application projects do not actually track integration tasks separately from other aspects of the development and operation of a new application system, so the cost of integration (whether 35 percent, 10 percent or 70 percent) is typically unknown by managers in both the IS organization and in the user departments.
It is important to realize that integration, for its own sake, is never really the goal. The purpose of integration is to support one or more development projects that will install or modify application software to accomplish tangible business goals. Integration middleware is infrastructure, like the plumbing in a house. A person is only motivated to upgrade the plumbing in a house if he or she is installing a new fixture (e.g., a new washing machine) or if the old pipes break entirely. Similarly, no enterprise will upgrade its integration mechanisms just to clean up an undocumented, chaotic set of interfaces. Enterprises will only be motivated to even consider new ways of integration when they have a major new application (or change to existing applications) that will require significant spending on integration. At that point, they may consider which style of integration to pursue.
Traditional versus Systematic Integration
For the remainder of this document, we assume that the enterprise has evaluated the overall business benefit of a proposed IS project and decided to proceed. So the interesting question about application integration is to decide whether the integration work should be done as part of the application development effort using traditional development practices and tools; or if a new-style, systematic approach to application integration is cost-justified. In the former, there is no separate "integration project" because integration is a subset of traditional development. In the latter, integration is carved out as a semiautonomous project that serves the needs of one or many, current or future, application projects.
In a traditional approach, application integration is the responsibility of each, separately managed application team. Developers do just enough integration to get their own application and business process changes to work. There is no broad-based, cross-application planning or documentation and there are no shared tools. Interfaces are point-to-point, going directly between the new application and other application systems, whether these are within the same business unit or elsewhere. For the purposes of this article, we are using the term "interface" in a very general sense, meaning any type of automated interaction such as file transfer, reading or writing files or databases, method or call entry points, and message-oriented middleware (MOM) and event interfaces. Traditional integration uses common development languages and middleware exclusively.
Newer, systematic approaches to application integration differ from traditional integration in two respects. First, the responsibility for coordinating some aspects of integration is delegated to a shared integration competency center (or "central integration team" or "center of excellence") rather than dispersed within each of the application project teams. Secondly, there is a shared integration infrastructure, generally including new forms of integration middleware such as an integration suite and/or business process management (BPM) software. These are actually independent decisions with separate paybacks, i.e., an enterprise can choose to do one or both of the organizational or software aspects. In practice, these are often done together under the umbrella of an "application integration project." However, as described above, a systematic application integration project is never initiated as a stand-alone decision. It is authorized only when the first major new use for it has been identified (except in rarest of circumstances). The first application provides much of the cost justification and shapes the form of the integration project.
The Potential Benefits of Systematic Integration
At the risk of over-simplifying a complex subject, the potential benefits of systematic integration are derived by doing one or more of the following basic activities:
using integration broker middleware,
using business process management middleware, or
instituting a competency center to coordinate integration work.
The case for each of these activities is outlined here (costs are discussed in the following section):
1. Integration Broker
Systematic integration may use an integration broker (e.g., Ascential/Mercator's Inside Integrator, IBM's WebSphere Business Integrator, Microsoft's BizTalk, Sybase's E-Biz Integrator, Tibco's Businessworks, Vitria's BusinessWare, or WebMethod's Integration Platform). These products supply two fundamental services, transformation and connectivity services such as intelligent routing. An integration broker is generally just part of an overall integration suite that also contains message-oriented middleware, Web services, support, adapters, adapter-building tools, a BPM engine and other components (however, in this section of the report, we are focusing just on the broker component). The transformation component of the broker automates the translation of data from one format to another, thus reducing the need for coding such translation logic in a sending or receiving application system. The connectivity and routing aspects of these products remove the burden of selecting the appropriate destination from the sending application programs. They also offload from the application program the connection logic (dialog or application-level flow logic) and make it easier to incrementally add, delete or move application systems. The value proposition of a broker is similar to that of a 4GL or many other software tools: it is faster and easier to implement these functions using a high level, graphical tool that is built for the task than it is to code the same logic in a 3GL by hand. Elsewhere, Gartner has published a methodology for calculating the extent of these broker benefits. Case studies have indicated that brokers can provide a 25 to 43 percent reduction of development effort (and thus cost), where simple interfaces achieve 25 percent savings and complex interfaces achieved 43 percent improvements.
Brokers deliver good payback for the transformation chores when:
there are many different messages or files to transform (dozens or hundreds of different documents, transaction types, methods, messages or database tables); or
the interface schemas are complex (hundreds or more fields per document or message, including repeating groups, nested structures, and numerous syntactic or semantic translations to perform, involving rules that would be hard to code in a 3GL); or
some interfaces will change or be added relatively often (weekly or monthly).
Brokers deliver good payback for the routing and connection chores when:
a single message (or file, table or other interface) is of relevance to more than one destination (e.g., an address change event is to be delivered to multiple receiving application systems, not just one);
the participating applications will be modified, moved, added or deleted frequently (putting new applications into the mix monthly, weekly or even in the course of a day without bringing down or changing the other systems);
routing rules are dynamic or complex ("send all orders that exceed $ 10,000 to the warehouse application in Chicago unless it involves one of the following eight types of goods or unless the order originated in Atlanta and is for delivery in Tampa next week" -- an expression readily implemented by the rules capabilities in most brokers but messy to build and maintain in regular 3GL code).
If none of these factors are present, then traditional middleware may be equally effective and a broker may be costly and complex overkill.
2. Business Process Management
Systematic integration may also use BPM software. These products help developers change the notion of application integration from one of managing interactions (e.g., making routing decisions) to a more sophisticated perspective that involves coordinating an entire multistep business process from end to end. They track and direct each instance of a business process, such as each individual order or medical insurance claim, through a life cycle that may consume seconds, minutes, hours, days or weeks. Unlike simpler forms of flow automation, BPM software "remembers" (i.e., maintains context information in a persistent file or database) for the duration of process instance that spans many individual activities.
Many integration middleware vendors assert that BPM is the major payback of their products, far outweighing the broker benefits of transformation and connectivity. Processes that are managed by BPM software are:
explicitly documented, which makes them better understood, more easily tuned, and more easily communicated among business and technical people;
represented in a graphical form so that nontechnical business people can design or change them;
easier to change as new business requirements arise; new steps and applications added or old ones dropped; or data flows or logic flows rearranged (thereby making the business itself more agile and better able to respond to business changes);
able to be monitored better, e.g., rule-driven thresholds may be designated so that alerts or alarms can be inserted (for example, "send me an e-mail with the transaction details if any order becomes more than one day late in any step of its fulfillment life cycle").
The business impact of these benefits will vary across industries and for different processes within each industry, but can be very large. A full exploration of this is beyond the scope of this report however.
3. Central Coordination of Integration Activities
Many of the potential benefits of systematic integration come from changing the IS organization structure and the procedures for developing and maintaining the interfaces (regardless of whether brokers or BPM tools are used at all). Integration was traditionally seen as the responsibility of individual application development teams which inevitably led to point-to-point interfaces, redundant development work and duplication of software tools. Decisions were locally optimized but globally suboptimized.
Systematic application integration recognizes that integration is an enterprise issue, spanning applications in multiple departments, data centers, remote offices and even in external trading partners. This realization drives IS management to create a new IS function, a shared integration competency center. This team has two aspects which may report together or separately:
The operational side installs and maintains a consistent software integration infrastructure. It ensures a 24x7 "enterprise nervous system" middleware stack on top of the corporate network, using some combination of communication middleware (e.g., messaging systems), platform middleware, integration middleware (e.g., brokers, gateways, and BPM software) and file transfer services. This is a technical support function, and may report to a system programming, system administration or network support business unit.
The development and application maintenance side collects and maintains documentation (metadata) for the application interactions (interfaces), and helps the developers in each of the individual application project groups build their connections into the integration infrastructure. This also may includes defining canonical document descriptions (e.g., using XSD), event descriptors, Web service interface definitions (e.g., WSDLs, stored in UDDI directories), method signatures or data models that will be used across multiple applications. These people generally act as mentors, guiding the developers associated with each application as they implement adapters. This function is akin to data administration and database administration (and may be combined with such groups).
The integration competency center is a service function, acting on behalf of multiple, semiautonomous application teams and business units. Its potential benefits are to:
reduce the development effort needed to document and code interfaces between systems by supplying methodologies, best practice knowledge and common adapter development tools and gateways;
reduce redundancy in the integration middleware. The enterprise will have fewer file transfer, MOM, broker, gateway, BPM and other middleware products, and thus lower software license fees and less need for trained technical personnel;
shorten the time it takes to add or change applications and their connections with other applications;
facilitate the re-use of Web Services interface definitions (e.g., documented using WSDL), business object document standards and other interface definitions (e.g., so that the enterprise has only one XML-based standard for a purchase order document, rather than numerous inconsistent newly-defined XML standards for the same thing).
Ultimately, the integration competency center and its technical deliverable, the intelligent "enterprise nervous system" network, may grow to serve the entire enterprise. However, no enterprise has gotten this far yet, to the best of our knowledge. We expect that such integration centers will not support even half of all of the integration connections in most large enterprises until after 2010. Competency centers start by covering just the applications and business units associated with the first relevant application and (sometimes) then grow. The extent of their benefits grow as the number of applications that are under their umbrella grows. More information about the role of an integration competency center is available elsewhere in Gartner research.
Estimating the Costs of Systematic Integration
Systematic integration may entail incremental new spending in the following areas:
Software license fees. A suite of integration middleware, including integration broker, BPM engine, related development and management tools, and pre-built adapters, typically costs between $ 200,000 and $ 400,000 from vendors such as Ascential (Mercator), BEA, IBM, Microsoft, SeeBeyond, Sybase, Tibco, Vitria or webMethods. Prices range widely, of course, based on the number and type of server computers, the need for backup and fail over systems, the pricing and discounting practices of each vendor, and many other factors. A basic broker without prepackaged adapters configured for a single PC can cost as little as $ 45,000. At the other end of the scale, a high end configuration for a complete system covering mainframes, several data centers, numerous Unix and Windows servers may cost more than $ 2,000,000. These fees are above and beyond the license fees for other software (operating systems, DBMSs, development tools, etc.) that would be used in both traditional or systematic integration approaches.
Software maintenance. Annual maintenance fees for integration broker suites and related middleware generally are set at 18 to 20 percent of the initial license fees (buyers should ensure that the maintenance fee is not calculated on the then-current, full list price of software; it should be calculated on the discounted level agreed upon with the first purchase agreement). Again, these fees are an incremental expense over the cost of traditional integration.
Consulting services. Many integration projects involve on site help from the broker middleware vendor and a general external services provider (ESP, for example, the Big Five all have practice areas aimed at application integration projects). The broker middleware vendors garner between 20 and 50 percent of their revenue from service business. These vendors often just provide training and assistance for the use of their own software, which are services needed at the beginning of the project. For example, a deal may involve one or two weeks of consulting time (at $ 250 per hour in the US) from the integration middleware vendor. The work of the ESP is likely to be much more extensive and costly, especially if it is a prime contractor involved in the entire application project from requirements definition through deployment. In a representative situation, external services may be 5 to 10 times the cost the integration software (e.g., 8 times $ 500,000 = $ 4,000,000). However, this figure is not really the cost of the application integration project, per se. It is the cost of external consulting services for building a new application solution, of which some part is the initial deployment of an application integration infrastructure. The integration infrastructure is usually destined to be used by subsequent application projects.
Enterprise IS people involved with the new forms of middleware will need training and time to learn. This staff time constitutes another incremental expense associated with the systematic integration project. Developers will spend two to five weeks of dedicated person-hours (spread over the first two to four months of the project) learning how to use broker or BPM software. A typical integration project may require training three to six people from the staff of the integration competency center (if there is one). These are mostly one-time costs, although some further spending will be required when a new person is brought in. This cost is incremental, compared to traditional integration, because it is driven by the additional software. Nevertheless, the overall labor costs of systematic integration may be still be lower for complex applications when the benefits of integration (listed above) are considered.
Ongoing technical support associated with the new software. Brokers, BPM tools and related software may require .25 to 2 or more full-time equivalents (or "headcount") for ongoing technical support, not counting new development work. Large environments with multiple brokers and often-changing applications will require more than 2 administrators. These efforts are incremental to other technical support efforts that would be required to support traditional approaches to integration (e.g., MOM administrators, network administrators, data base administrators (DBAs)). However, this labor may be more than offset by a reduction of effort within the application development groups if the same level of integration was to be done using traditional methods (a big "if" however).
Integration competency center personnel. A competency center often consists of four to six people on a permanent basis (we have seen teams range from .5 people to more than a hundred people, although the larger groups are responsible for many issues not specifically related to systematic integration such as MOM and some of the direct, hands-on coding for applications that were tied into the integration infrastructure). Roughly half of the work done the competency center is attributable to the training and technical support of the incremental integration middleware (described in the previous two points). The other half of the competency center costs are related to development activities such as maintaining the "city plan," which includes documentation for the application interactions (interface metadata). The former effort is required if the enterprise is using brokers or BPM tools; the latter is required by any integration competency center since interface documentation is an essential aspect of the function of the center (it is the key enabler for re-using the interface definitions among multiple different application projects).
In general, the hardware and network costs of systematic integration will be roughly the same as the hardware and network costs of traditional integration in most cases. In theory, systematic integration should reduce the number of redundant messages and file transfers, so the network and system overhead should decrease. In practice, however, enterprises do not seem to experience measurable gains in this regard. The re-use or "sharing" of messages and file transfers is not very high because even a systematic solution ends up sending the same data multiple times anyway. Or, there is only one application that wants to get the data so there is no opportunity for re-use in the first place. Furthermore, an integration broker usually runs on its own dedicated server or servers, although it sometimes will share a server with one of the participating application systems. Again, in theory, systematic integration should decrease the processing overhead of the application system computers (thus allowing for smaller and cheaper application servers) because the work of transformation and routing is offloaded to the integration broker servers. However in practice, such gains are offset by the incremental costs of the servers that run the integration brokers (plus any development, testing or failover servers). In a medium or large enterprise, the cost of these additional servers is very small compared to overall hardware costs, and almost unmeasurable compared to total cost of ownership (IS staff, software, environmentals etc.). The net result of all of these factors is to make hardware and network costs a non-issue in the choice between systematic vs. traditional integration. However, for budgeting purposes, planners must expect that either form of integration will entail network and system overhead, in addition to that used to run the applications themselves.
Analyzing IS Cost Reduction
Some managers like to use "hard savings" within the IS budget to justify an investment in systematic integration. However, this approach can be problematic because there is only a limited amount of statistical data available on this subject and most of it is insufficiently detailed and verified. Most integration projects are not actually approved based on hard savings in the IS department, and when they are, the hard savings are not measured after the fact in most cases. It is better to justify a systematic approach to integration based on business benefits rather than on reductions in the IS budget (see next section).
Nevertheless, it is possible to analyze the cost avoidance in the IS department associated with integration brokers, BPM products and integration competency centers and draw some general conclusions. These cost reductions should be considered as three separate issues because each can be pursued independently if desired (although they are executed together more often than not):
Integration brokers represent a shift in labor costs, not an additional new job. For example, the logic related to functions such as transformation and connectivity are pulled out of the application program and run in the broker. The question then is whether the cost of the additional integration middleware (including maintenance fees, training costs, and administrative labor) is offset by the reduction in labor costs in the application development team. For small applications with a limited number of interfaces, the answer will be no. Integration brokers will pay off only when there are many interfaces, or the interfaces are changing frequently so that the fixed broker costs (software fees and the developers' learning curve) can be spread more broadly. A worthwhile broker project usually involves at least two dozen interfaces (and sometimes many hundreds) and several application systems (typically three to five for an initial application). As described above, brokers generally deliver 25 to 43 percent savings in interface development time and expense through the use of transformation facilities and adapter development toolkits (ADKs). Savings can be twice as large (50 to 80 percent) if a pre-built adapter for a particular packaged application is available because the cost of building that adapter will be substantially reduced. In other words, if there is an off-the-shelf adapter for a particular process and a particular packaged application, it can be significantly faster and cheaper to buy it and tailor it than to develop a similar connection from scratch in a traditional 3GL.
BPM tools cannot be justified through simple, direct IS cost avoidance. The IS staff and budget will rarely if ever be reduced. The benefits of BPM appear outside of the IS department in the form of improved business processes. The IS labor and software costs in a business process implemented with BPM tools will actually be higher in a simple accounting terms than similar costs in a process that does not use BPM software. However, the process implemented through BPM is likely to be superior to that implemented with standard tools.
A well-run integration competency center will have a positive payback for almost all medium- or large-scale of integration scenarios, regardless of whether any commercial broker or BPM tools are used. The competency center is usually staffed by IS people transferred from other parts of the organization, and therefore rarely involves any net new headcount. Most of the activities of this team are tasks that would have had to be done by the application groups in traditional integration anyway (e.g., setting up and running the MOM, gateways, Web services tools, file transfers and any other middleware). It must be noted that a few of the functions of the competency center are new, incremental effort. In particular, the competency center will document interfaces, including Web Services, batch file schemas, and everything in between, in a more visible and widely accessible fashion than a traditional development team usually documents its interfaces. Nevertheless, the long term payback from explicit, centralized documentation is clear and substantial. It enables interface reuse for subsequent applications and lowers support and modification costs. More importantly, developers can build new applications more quickly even if they are not re-using existing interfaces by have good documentation on what is running in production today. Just as managers readily accept the notion of having a common enterprise network support group and a centralized PC help desk, the justification of using a common integraiton team that can share integration expertise and documentation is common sense. A competency center also plays a key role in reducing the redundancy of middleware technologies. No enterprise is well served by having more disparate MOM, integration brokers, BPM products, or B2B integration servers than necessary. Of course, organizational politics often complicate the implementation of the competency center (more explanation of this is available elsewhere in Gartner research). A competency center is the foundation for a long range integration strategy wherein additional applications will be added to a common infrastructure over time. A small competency center (less than one full time equivalent) is helpful for environments of modest size. The only circumstance in which a competency center is overkill is where there are very few interfaces (e.g., less than 24) and few application systems (three or less); where these dimensions are not expected to grow; and where the information exchanged has no relevance to other current or future applications in the enterprise.
Analyzing Business Benefits
In practice, systematic integration brings business benefits outside of the IS department that are larger, more compelling and more certain than the returns that are derived within the IS budget. These overall business benefits may be either 1) hard savings in projected spending (e.g., net cost reductions and margin improvements based on productivity gains of clerical staff or professional staff) or, more likely, (2) there may be enhancements in strategic effectiveness (e.g., long term revenue growth, customer satisfaction, or competitive positioning).
Most integration projects today are authorized not to save IS costs, but rather, to enable the development of a demanding new business process that must span multiple application systems. Enterprises are under pressure from a variety of external forces to implement integrated processes that complete more quickly or provide more data than previous processes. This pressure may come from trade associations, B2B business partners or legal requirements. Examples of such pressures include the HIPAA regulations (for health care information reporting and insurance claims), new SWIFT protocols and message types (for financial transactions), Sarbanes Oxley (disclosure requirements for most U.S. businesses), UCCNet (for retail distribution) and RosettaNet (for high tech manufacturing). Other integration projects are compelled by business needs for high quality, near-real-time data collection and dissemination that arise from customers (e.g., supplying up-to-the-minute data through a customer portal), employees (e.g., self-service employee benefits information through an employee portal) or business partners (e.g., consolidated bills, statement and order state information).
In theory, any of these integration projects could be built using traditional tools and custom code and the same business benefits could be achieved. But in practice, many such solutions can be built faster using a systematic integration approach, and the result is more flexible and amenable to further change than if the same process was implemented with traditional approaches. When there are external legal or business needs to implement a new business process of this type, management does not debate whether or not to pursue the project. It must proceed with the project and the only questions revolve around which kinds of integration middleware to use and how to organize some form of integration competency center.
For further information, please contact the QuickPath inquiry center at Gartner Inc. at firstname.lastname@example.org.
About the Author
W. Roy Schulte is Vice President and Distinguished Analyst at Gartner Inc. in Stamford Connecticut. Mr. Schulte was a co-author of the 1996 report that introduced the term SOA to the industry. He also originated research into the field of message brokers, coined the term business activity monitoring (BAM), and wrote the first analyst reports on the zero-latency enterprise and the enterprise service bus (ESB). His current work centers on event-driven architecture (EDA). Mr. Schulte has more than 20 years of industry experience spanning user enterprises, IT vendors and Gartner Inc. He holds a BS from the Massachusetts Institute of Technology and an MS from MIT's Sloan School of Management.
Gartner is a research and advisory firm that helps more than 10,000 clients leverage technology to achieve business success. GartnerŐs businesses consist of Research, Consulting, Measurement, Events and Executive Programs. Founded in 1979, Gartner is headquartered in Stamford, Connecticut and has over 3,800 associates, including approximately 1,000 research analysts and consultants, in more than 75 locations worldwide.