As integration becomes more complex and far reaching, the need to make sense of the hundreds, sometimes thousands of data stores within an enterprise or trading community is becoming critical. Indeed, corporate data typically exists in many different types of data storage technologies and schema types. Creating a shared view of the information that can be used for any purpose is essential in both establishing a strategic integration infrastructure and making business decisions using live, meaningful corporate data.

We’ve seen the rise of data abstraction middleware designed to place any number of enterprise databases, sometimes APIs (such as SAP’s BAPI), flat files, and even Web services, behind a database abstraction layer. This is really nothing new; we’ve been creating virtual databases in middleware for years. However, the newer capabilities of certain vendors have made this type of technology carry an enhanced value proposition, and this type of solution will find its way into many enterprises over the next year or two.

Of course, any new technology has to use a new buzzword and for this space it’s EII (Enterprise Information Integration). This is also known as virtual databases, also known as data federation. For our purposes let’s call it what it really is…data abstraction. Data abstraction layers, and the technology that provides this mechanism, could indeed become one of the most important integration tools in the arsenal, making the use of many heterogeneous data stores and databases much easier to deal with, and thus making the data and the databases a more valuable and more strategic resource for the enterprise.

The notion of data abstraction layers follows a few basic principles:

First, it’s better, less risky, and less expensive to leave the data where it is, instead of storing it redundantly and representing it differently depending on the required pattern of use. The idea of data abstraction is to leave it be and make it useful in other, less invasive ways. This is both a cost and a scalability issue; to the extent possible, we would like to adhere to the principle of “one fact in one place.”

Second, following from our first principle, we want to present data to applications in a way that may be very different from how it is stored. To do this, we need to provide a flexible mechanism, allowing the data view (represented schema) to change based on how the data needs to be seen. We can represent one or many physical schemas in many different ways, depending on the need. It should be possible to create hundreds of views customized for hundreds of different purposes, even reusing views from interface to interface, purpose to purpose.