How Will BI Solutions Evolve to More Easily Handle Distributed Data Processing (DDP)?

Hadoop, NoSQL, Key Value Stores, etc, are storage and/or grid friendly parallel processing data management designs. Today, by themselves, most do not have the rich calculation capabilities or array processing capabilities of a SQL, MDX or other query and BI modeling languages. What they can do is search across massive, high dimensional unstructured data without predefined analytical models. In certain "BI" applications that simple search may be enough - in other applications simple "search" may fall short.

How data from these distributed sources will be federated, integrated and modeled will likely fall to a new generational of data management technologies designed to support traditional MDM concepts with emerging semantic web approaches that will allow data to be reported or staged for further processing. In the future analytic functionality will become more independent to the underlying storage and index models.

It is actually very real, very cool and promises to be game changing.

I agree with at least two of Dyke's key points: 1) that future analytic functionality will become more independent to the underlying storage and index models, and 2) that DDP is very real, very cool and promises to be game-changing.

Further, I believe that a well-architected (modern) BI tool will add quite a lot of value to the world of big, distributed data. True to another of Dyke's points, the key will be semantics, and within a BI tool, this means metadata. Properly designed, a BI tool should be able to connect to big data sources, query for data that can then be described semantically (even held alongside structured data), and viewed then manipulated within a dashboard or report.

The insight gained at the intersection of traditional, structured data AND unstructured "big" data sources will be substantial and will set the pace for the future of business intelligence.

Brian Gentile
Chief Executive Officer
Jaspersoft

DDP (Distributed Data Processing) is an idea whose time has come (a la Victor Hugo). Given the paradigm shift towards 'Big Data' for many organizations, it is becoming clear that the 'divide & conquer' strategy is a sensible, practical way for data management. Technology like Hadoop, DataRush etc. provide the platform for distributed data processing over MPP & SMP architectures. With ever increasing CPU cores, clock speeds, RAM sizes, I think we have the power available to store & process huge volumes of distributed data.

The bigger question is - "How do we make sense of all this petabytes & exabytes of data". I don't have an answer but let me share some perspective: (at the risk of sounding irrelevant!)

1) Ebay, Google, Walmart, Facebook etc. manage petabytes of information and seems to be making sense out of them - analytically speaking. So there's hope for others too.

2) Recently read about the A2DB (Advanced Analytic database) from Algebraix data. I think such data models have a play in the Big Data world.

3) CMIS specifications and the whole world of Content management systems will participate in DDP.

4) There are companies like ClickFox that creates specific models for IVR logs, weblogs etc. and provides analytics for multi-channel customer experience.