Successful business intelligence implementations can be a real drag.

After all, the successful departmental data mart attracts other parts of your organization. They want to get in on the act – never mind the fact that you didn’t model your data to meet their requirements, or that the reports you painstakingly crafted use different definitions of key terms, or that their data lives in a completely different set of applications.

It’s hard enough to manage a data integration project successfully the first time. In order to get our first project out the door, we usually use two common development practices: effective up-front planning, and rapid prototyping and user-testing of small, manageable project chunks.

In a business intelligence context, up-front planning involves modeling an enterprise data warehouse or defining a set of dimensions that can be used in any data mart. Unfortunately, this planning often runs against the grain of the iterative, prototype-and-test method that we’d like to use – and although both methods get our first project into production, neither one really solves the unexpected additional challenges that our success brings to us.

That often makes us see the data as somebody else’s problem: we require the appropriate data in order to generate our data marts, our reports, and our cubes, so we define the model first and then let the extraction, transformation, and load (ETL) team pull the information from the systems of record. When the boss, finally pleased with the data warehouse results, says she wants to add a particular set of numbers, we jump through hoops to get the data into the warehouse as quickly as possible – and pray that we can fit it within our current model, or use our mutually conformed dimensions to create the necessary reports.

In other words, we struggle to turn our initial good business intelligence projects into a single, healthy business intelligence environment. We approach each new BI problem as a unique data integration problem, a balancing act that pits BI user demands against constraints imposed by ETL and operations staff. We use different tools, different models, and different techniques to integrate the data, leading to considerable waste and quality issues. Often, sensing the difficulties that heterogeneity brings, we ensure that all reports come out of an operational data store, data warehouse, or data mart – even when our users want operational data. Finally, we reconsider governance issues at each turn, which leads to redundant – not to mention confusing, and therefore vulnerable – security policies and technologies for data access, data quality, and audit trails.