Enterprise Information Integration (EII) is proving to be an amazingly effective
vehicle for traveling to the new space known as Service Oriented Architecture
As modern-day technology navigators, SOA architects are finding that EII or
data federation provides an excellent charter for their voyage from planning
and design to implementation. Part of this charter is asking and answering the
Where does Data Federation/EII fit in the world of SOA?
How are EII-based data services overcoming the biggest hurdle in new SOA
Why are data virtualization, data abstraction and on-demand data integration
services now critical capabilities required in every SOA?
Who is gaining value from these new capabilities today?
How does one get started on a similar journey?
EII - Data Integration based on Data Federation
According to Colin White, president of Oregon-based industry analyst firm BI
Research, "Data Integration provides a unified view of the business data
that is scattered throughout an organization. This unified view can be built
using a variety of different techniques and technologies." *1
These techniques may include:
Federation - a virtual federated view of disparate data assembled dynamically
at data access time; and
Consolidation - a physical view of data captured from multiple disparate
data sources and consolidated into an integrated data store like a data warehouse
or operational data store; and,
Propagation - a propagated view of data created by synchronizing data from
one database to another such as product data across manufacturing, supply
chain and order management systems.
Data-integration technologies have evolved to support each of these techniques
Enterprise Information Integration (EII) for federating data
Extract, Transform and Load (ETL) for consolidating data
Enterprise Application Integration (EAI) for propagating data.
In the same article, Mr. White defines EII in more detail. "EII provides
a virtual business view of dispersed data. This view can be used for demand-driven
query access to operational business transaction data, a data warehouse, and/or
unstructured information. EII supports a data federation approach to data integration.
The objective of EII is to enable applications to see dispersed data as though
it resided in a single database. EII shields applications from the complexities
of retrieving data from multiple locations, where the data may differ in semantics
and formats, and may employ different data interfaces."
SOA - No Agility Without Data
SOA is all about business and IT agility. SOA seeks to achieve this agility
primarily through interoperability and reuse. Yet, interoperating with and reusing
existing data are the biggest bottlenecks in deploying SOA.
Most new applications, SOA or not, build on data from an existing systems foundation.
This data is complex, diverse and spread across the enterprise in various technology
and application silos. For example, a new customer service application might
include current billing data from packaged applications such as SAP; historical
transaction data from a data warehouse; self-service customer portal data in
the form of XML and HTTP files; and so forth. Each source has its own access
mechanisms, syntax, security, etc., and few are structured properly for reuse.
All these factors combine to slow down development projects. All these factors
Data services are a new class of SOA service that helps the IT team to overcome
these bottlenecks. For many, the term "data services" is new; yet,
the concept is simple. Data services span the gap between how and where existing
data is stored, and how and where the data is used so all data appears as if
it's in one place. According to Boston-based industry analyst firm AMR Research,
"No SOA plan is complete without a data services layer." **2
Data Services - Where EII and SOA Come Together
Data services are the next step in the federated approach to data integration.
And the data services layer--known as information servers and provided by best-of-breed
specialists such as Composite Software and SOA stack providers such at IBM--are
the next-generation data federation technology beyond EII.
EII's roots are clearer by comparing Colin White's definition of EII above
with key data services capabilities as follows:
Data Virtualization makes all data logically "appear" as if it
is in one place, rather than where it is physically distributed in the siloed
data sources that have proliferated over the years.
Data Abstraction simplifies complex data by transforming it into useful
views and services that applications-level developers and report writers can
On-Demand Data Integration enables the combination or federation of data
from all different sources into something that is more meaningful. Further,
it delivers this data to a wide variety of solutions in real time.
Common terms such as "virtual," "on demand," "federate,"
"complexities," and others make this common ground obvious. However
the primary difference between 2005 EII and 2007 Data Services is how data services
have been implemented in support of SOA.
Whereas EII was originally and primarily a relational paradigm, data services
can also handle SOA paradigms such as hierarchical XML with equal aplomb. Data
services within the data services layer can be relational views or SOA-compliant
services. And data services products leverage SOA standards such as WS-I for
security and SOAP for interfaces, to name a few.
Architecturally, SOA allows for the combination of data virtualization, data
abstraction and on-demand data integration services into a unified layer. This
data services layer can then support a complete range of consuming solutions
including dashboards, reporting, analytics, composite applications and more,
as well as a complete range of data sources including packaged applications,
databases, warehouses, Web services and others. See Figure 1.
This architecture, with its three-tiered, loose-coupling approach has many
benefits. For example, loose-coupling gives you the flexibility required to
deal with each layer in the most effective manner, as well as the agility to
work quickly across layers as you change applications, schemas, or underlying
Delivering Value Today
Data services enable the business and IT agility that SOA promises. Data services
break the data integration bottleneck that arises from the data complexity and
silos characteristic in today's enterprises. Data services help IT respond faster
to business requests by delivering the data to your new business solutions require
faster than ever before. Further, data services reduce costs by reusing not
only existing data in new and powerful ways, but also reusing data services
across multiple projects.
The Value of Data Virtualization:
Enterprises today have multiple systems of record, multiple complementary applications,
multiple external data sources, and future plans for even more. Yet, with most
new data requirements spanning these traditional silos, it is little wonder
that accessing data from across them is so challenging. Data virtualization
services let you virtualize your data silos, so that all data logically "appears"
as if it is in one place. Some examples include:
Financial Research Data Services Layer - Data services form a middle layer
between a multi-terabyte financial research database and a variety of Matlabs
analytical and custom financial engineering applications.
Reporting Data Services Layer - Multiple reporting requirements including
Prime Brokerage, Reconciliation, Risk Management, etc., share a common set
of data services.
Scientific Research Workbench - This scientific workbench leverages data
services to access research, clinical trial, FDA submission data, and more.
IT Performance Management Dashboard - Gathering SLA and other system status
metrics from a range of application management systems, data services give
IT operations managers a complete performance picture.
The Value of Data Abstraction:
The mismatch between how data is stored (formats, structures, APIs, etc.) and
how it will be used is more pronounced than ever. For example, many Web-based
applications assume hierarchical XML data, but the source data may be stored
in a tabular relational data store. Data stored by SAP uses SAP-specific data
structures. Accessing this data requires SAP-specific APIs and deep SAP data
model knowledge and skills.
Data abstraction services abstract complex data, resolving structural and semantic
issues to overcome this mismatch. Data experts can model the data and then build
data services in the form of Web services or SQL tables that provide the exact
data required. Applications teams can then develop new applications using these
rationalized views of the data, instead of forcing the teams to learn and keep
up with all the complexities inherent in the source data itself.
Single View of Fitness Club Member - Data services abstract customer data from
a range of technologies including their custom member management system, Siebel
CRM, Excel-based personal trainer data, and Web-services-based spa management
Payroll Variance Analysis - Data services abstract budget data stored in a
central budgeting application and actual payrolls managed by several SAP instances
so that developers can deliver an advanced payroll variance analysis application.
Insider Trading Compliance - With SEC reporting mandated and fines for non-compliance
harsh, data services simplify required data, so that applications developers
can provide the required compliance reports.
The Value of On Demand Data Integration:
Combining data from different sources gives new life to isolated enterprise
data. Traditionally, this integration was done in batch mode via ETL. But yesterday's
data tomorrow may not to be good enough. And not every consuming application
is architected to receive data via an ODBC connection to a data mart, for example.
On-demand data integration services provide information on-demand, when and
how it is needed. Data services can be push-based, with information updates
occurring periodically, driven by event triggers. Or they can be pull-based,
or called on demand by the data-consuming application. Frequent queries may
be cached as a means of balancing timeliness of delivery with timeliness of
Interest Rate Sensitivity Analysis - This analytic application uses data services
to deliver up-to-the-minute interest rates and financial position data.
Virtual Month-End Close - Data services federate the latest accounting data
from a number of on-line accounting systems enabling financial metrics to be
calculated at any time during the period.
Global Inventory Balances - Inventory balances can be calculated at any time
using data services that combine on-hand inventories across manufacturers, shippers,
distributors and retailers.
Achieving the SOA Mission, One Integration at a Time
To succeed with data services, it's best to start small with a focused team,
on a focused set of projects, for a single department or line of business. On
these initial projects, be sure to track tangible time-to-solution and costs
savings. This allows for self-funding additional licenses and other required
resources as the projects evolve into enterprise-wide deployments.
In selecting an information server, consider ease of use, performance, and
openness as important criteria. For enterprises invested in popular packaged
applications such as SAP and Oracle, consider vendors that provide pre-built
data services out of the box to help jump start those SOA-integration projects.
During the development of data services, keep reuse in mind, but don't let
it slow down individual projects. It is better to build the new services, project
by project, thereby making them available to other projects in the data services
layer. Developers can then choose whether to reuse or extend an existing service
or to build a new one from scratch. And over time, the mix will change, with
reuse becoming a greater percentage of the work.
SOA means agility. But achieving agility is difficult-if not impossible-without
first overcoming the SOA data integration challenge.
The federated approach to data integration instantiated through EII technology
has proven to be an effective path to data integration success. Data services
that provide data virtualization, data abstraction, and on-demand data integration,
along with new technology known as information servers, are the next generation
of data federation and EII.
Robert "Bob" Eve has held executive-level marketing and business development roles at several leading enterprise software companies prior to joining Composite Software. At Informatica and Mercury Interactive, he helped penetrate new segments in his role as vice president of Market Development. Eve ran Marketing and Alliances at Kintana (acquired by Mercury Interactive in 2003) where he defined the IT Governance category. As vice president of Alliances at PeopleSoft, Eve was responsible for more than 300 partners and 100 staff members. In his nearly ten years at Oracle, he held roles such as Oracle Manufacturing's first product manager and head of Oracle's revolutionary software integration program, CAI. Eve has an MS degree in Management from MIT and a BS with honors in Business Administration from University of California, Berkeley. He is a frequent contributor and speaker on data virtualization.