Using EII to Fast-Track Your Way to SOA

Enterprise Information Integration (EII) is proving to be an amazingly effective vehicle for traveling to the new space known as Service Oriented Architecture (SOA).

As modern-day technology navigators, SOA architects are finding that EII or data federation provides an excellent charter for their voyage from planning and design to implementation. Part of this charter is asking and answering the following questions:

  • Where does Data Federation/EII fit in the world of SOA?
  • How are EII-based data services overcoming the biggest hurdle in new SOA deployments?
  • Why are data virtualization, data abstraction and on-demand data integration services now critical capabilities required in every SOA?
  • Who is gaining value from these new capabilities today?
  • How does one get started on a similar journey?



EII - Data Integration based on Data Federation

According to Colin White, president of Oregon-based industry analyst firm BI Research, "Data Integration provides a unified view of the business data that is scattered throughout an organization. This unified view can be built using a variety of different techniques and technologies." *1
SOA

These techniques may include:

  • Federation - a virtual federated view of disparate data assembled dynamically at data access time; and
  • Consolidation - a physical view of data captured from multiple disparate data sources and consolidated into an integrated data store like a data warehouse or operational data store; and,
  • Propagation - a propagated view of data created by synchronizing data from one database to another such as product data across manufacturing, supply chain and order management systems.

Data-integration technologies have evolved to support each of these techniques including:

  • Enterprise Information Integration (EII) for federating data
  • Extract, Transform and Load (ETL) for consolidating data
  • Enterprise Application Integration (EAI) for propagating data.

In the same article, Mr. White defines EII in more detail. "EII provides a virtual business view of dispersed data. This view can be used for demand-driven query access to operational business transaction data, a data warehouse, and/or unstructured information. EII supports a data federation approach to data integration. The objective of EII is to enable applications to see dispersed data as though it resided in a single database. EII shields applications from the complexities of retrieving data from multiple locations, where the data may differ in semantics and formats, and may employ different data interfaces."

SOA - No Agility Without Data

SOA is all about business and IT agility. SOA seeks to achieve this agility primarily through interoperability and reuse. Yet, interoperating with and reusing existing data are the biggest bottlenecks in deploying SOA.

Most new applications, SOA or not, build on data from an existing systems foundation. This data is complex, diverse and spread across the enterprise in various technology and application silos. For example, a new customer service application might include current billing data from packaged applications such as SAP; historical transaction data from a data warehouse; self-service customer portal data in the form of XML and HTTP files; and so forth. Each source has its own access mechanisms, syntax, security, etc., and few are structured properly for reuse. All these factors combine to slow down development projects. All these factors prevent agility.

Data services are a new class of SOA service that helps the IT team to overcome these bottlenecks. For many, the term "data services" is new; yet, the concept is simple. Data services span the gap between how and where existing data is stored, and how and where the data is used so all data appears as if it's in one place. According to Boston-based industry analyst firm AMR Research, "No SOA plan is complete without a data services layer." **2

Data Services - Where EII and SOA Come Together

Data services are the next step in the federated approach to data integration. And the data services layer--known as information servers and provided by best-of-breed specialists such as Composite Software and SOA stack providers such at IBM--are the next-generation data federation technology beyond EII.

EII's roots are clearer by comparing Colin White's definition of EII above with key data services capabilities as follows:

  • Data Virtualization makes all data logically "appear" as if it is in one place, rather than where it is physically distributed in the siloed data sources that have proliferated over the years.
  • Data Abstraction simplifies complex data by transforming it into useful views and services that applications-level developers and report writers can easily understand.
  • On-Demand Data Integration enables the combination or federation of data from all different sources into something that is more meaningful. Further, it delivers this data to a wide variety of solutions in real time.

Common terms such as "virtual," "on demand," "federate," "complexities," and others make this common ground obvious. However the primary difference between 2005 EII and 2007 Data Services is how data services have been implemented in support of SOA.

Whereas EII was originally and primarily a relational paradigm, data services can also handle SOA paradigms such as hierarchical XML with equal aplomb. Data services within the data services layer can be relational views or SOA-compliant services. And data services products leverage SOA standards such as WS-I for security and SOAP for interfaces, to name a few.

Architecturally, SOA allows for the combination of data virtualization, data abstraction and on-demand data integration services into a unified layer. This data services layer can then support a complete range of consuming solutions including dashboards, reporting, analytics, composite applications and more, as well as a complete range of data sources including packaged applications, databases, warehouses, Web services and others. See Figure 1.

This architecture, with its three-tiered, loose-coupling approach has many benefits. For example, loose-coupling gives you the flexibility required to deal with each layer in the most effective manner, as well as the agility to work quickly across layers as you change applications, schemas, or underlying data sources.

Delivering Value Today

Data services enable the business and IT agility that SOA promises. Data services break the data integration bottleneck that arises from the data complexity and silos characteristic in today's enterprises. Data services help IT respond faster to business requests by delivering the data to your new business solutions require faster than ever before. Further, data services reduce costs by reusing not only existing data in new and powerful ways, but also reusing data services across multiple projects.

The Value of Data Virtualization:

Enterprises today have multiple systems of record, multiple complementary applications, multiple external data sources, and future plans for even more. Yet, with most new data requirements spanning these traditional silos, it is little wonder that accessing data from across them is so challenging. Data virtualization services let you virtualize your data silos, so that all data logically "appears" as if it is in one place. Some examples include:

  • Financial Research Data Services Layer - Data services form a middle layer between a multi-terabyte financial research database and a variety of Matlabs analytical and custom financial engineering applications.
  • Reporting Data Services Layer - Multiple reporting requirements including Prime Brokerage, Reconciliation, Risk Management, etc., share a common set of data services.
  • Scientific Research Workbench - This scientific workbench leverages data services to access research, clinical trial, FDA submission data, and more.
  • IT Performance Management Dashboard - Gathering SLA and other system status metrics from a range of application management systems, data services give IT operations managers a complete performance picture.

The Value of Data Abstraction:

The mismatch between how data is stored (formats, structures, APIs, etc.) and how it will be used is more pronounced than ever. For example, many Web-based applications assume hierarchical XML data, but the source data may be stored in a tabular relational data store. Data stored by SAP uses SAP-specific data structures. Accessing this data requires SAP-specific APIs and deep SAP data model knowledge and skills.

Data abstraction services abstract complex data, resolving structural and semantic issues to overcome this mismatch. Data experts can model the data and then build data services in the form of Web services or SQL tables that provide the exact data required. Applications teams can then develop new applications using these rationalized views of the data, instead of forcing the teams to learn and keep up with all the complexities inherent in the source data itself.

Examples include:

Single View of Fitness Club Member - Data services abstract customer data from a range of technologies including their custom member management system, Siebel CRM, Excel-based personal trainer data, and Web-services-based spa management application.

Payroll Variance Analysis - Data services abstract budget data stored in a central budgeting application and actual payrolls managed by several SAP instances so that developers can deliver an advanced payroll variance analysis application.

Insider Trading Compliance - With SEC reporting mandated and fines for non-compliance harsh, data services simplify required data, so that applications developers can provide the required compliance reports.

The Value of On Demand Data Integration:

Combining data from different sources gives new life to isolated enterprise data. Traditionally, this integration was done in batch mode via ETL. But yesterday's data tomorrow may not to be good enough. And not every consuming application is architected to receive data via an ODBC connection to a data mart, for example.

On-demand data integration services provide information on-demand, when and how it is needed. Data services can be push-based, with information updates occurring periodically, driven by event triggers. Or they can be pull-based, or called on demand by the data-consuming application. Frequent queries may be cached as a means of balancing timeliness of delivery with timeliness of data.

Examples include:

Interest Rate Sensitivity Analysis - This analytic application uses data services to deliver up-to-the-minute interest rates and financial position data.

Virtual Month-End Close - Data services federate the latest accounting data from a number of on-line accounting systems enabling financial metrics to be calculated at any time during the period.

Global Inventory Balances - Inventory balances can be calculated at any time using data services that combine on-hand inventories across manufacturers, shippers, distributors and retailers.


Achieving the SOA Mission, One Integration at a Time

To succeed with data services, it's best to start small with a focused team, on a focused set of projects, for a single department or line of business. On these initial projects, be sure to track tangible time-to-solution and costs savings. This allows for self-funding additional licenses and other required resources as the projects evolve into enterprise-wide deployments.

In selecting an information server, consider ease of use, performance, and openness as important criteria. For enterprises invested in popular packaged applications such as SAP and Oracle, consider vendors that provide pre-built data services out of the box to help jump start those SOA-integration projects.

During the development of data services, keep reuse in mind, but don't let it slow down individual projects. It is better to build the new services, project by project, thereby making them available to other projects in the data services layer. Developers can then choose whether to reuse or extend an existing service or to build a new one from scratch. And over time, the mix will change, with reuse becoming a greater percentage of the work.

Conclusion

SOA means agility. But achieving agility is difficult-if not impossible-without first overcoming the SOA data integration challenge.

The federated approach to data integration instantiated through EII technology has proven to be an effective path to data integration success. Data services that provide data virtualization, data abstraction, and on-demand data integration, along with new technology known as information servers, are the next generation of data federation and EII.

References:

*1 "Data Integration: Using ETL, EAI, and EII Tools to Create an Integrated Enterprise"
Nov 2005
Colin White, President, BI Research
Copyright 2005 BI Research, Inc.

**2 Successful Service-Oriented Architectures Build in a Data Services Layer
November 10, 2005
Eric Austvold, Jeff Hojlo
Copyright 2005 AMR Research, Inc.

About the Author

Robert "Bob" Eve has held executive-level marketing and business development roles at several leading enterprise software companies prior to joining Composite Software. At Informatica and Mercury Interactive, he helped penetrate new segments in his role as vice president of Market Development. Eve ran Marketing and Alliances at Kintana (acquired by Mercury Interactive in 2003) where he defined the IT Governance category. As vice president of Alliances at PeopleSoft, Eve was responsible for more than 300 partners and 100 staff members. In his nearly ten years at Oracle, he held roles such as Oracle Manufacturing's first product manager and head of Oracle's revolutionary software integration program, CAI. Eve has an MS degree in Management from MIT and a BS with honors in Business Administration from University of California, Berkeley. He is a frequent contributor and speaker on data virtualization.

More by Robert Eve