Overview

With the hype surrounding Service-Oriented Architecture, people often forget that SOA is mostly about plumbing, and that a substantial portion of the top-line business value of SOA is derived directly from the data it pushes around.

Today, the data universe of Business Intelligence (BI), Data Warehousing and Master Data Management (MDM) still lives largely outside the scope of SOA. But an enterprise view of Data Services may well bring these worlds together.

When properly architected and executed, Data Services can provide the link that unifies conventional data systems with the emerging SOA paradigm. By offering a decoupled data façade and an easily virtualized API, Data Services can give SOA the opportunity to establish system control.

Part 1: The Primacy of Data

It’s always been about the data. Decades of punditry about EAI, ETL, MDM and SOA lead us to the same conclusions -- data is king in enterprise software.

Sometimes the enterprise software sector loses sight of that simple reality. In the past 15 years, with the rise of Java, the hype surrounding EII, EAI and SOA, and the rise of XML, and quietly, the billions spent in ETL projects -- it's too easy to forget why we build and buy all that infrastructure. We do it for the data.

There has been more data created since 2000 than in all of human history preceding then. (Figure 1)

Figure 1: Information Explosion (Adaptive Information, Wiley & Sons 2004)

These trends show us that the data problem is getting worse, not better.

Enterprise infrastructure is surprisingly unchanged since the early 1990s when Message-Queues (MQ), Transaction Processing Systems (TPS), and ETL tools were really the backbone of enterprise software. Guess what? They still are. Despite the growing adoption rates of BPM, SOA, ESB, and EII – the MQ, TPS, and ETL backbones are still there.

The strain of all that new data and the demand for mature tooling has paradoxically made the existing, proven software infrastructure look pretty attractive. Most new XML-centric solutions for data can't scale to the mult-terabyte sized problem that is typical of a Global 2000 business. Thus, a knowledgeable architect will revert to the proven patterns of RDBMS as the backbone of a data architecture using MQ, TPS, and ETL interfaces as the pipes for pushing all that data around.