In this blog, I have been writing extensively in other entries about how new Business Intelligence tools need to be highly interactive and flexible for users to really benefit from their use.
If you need Insights from mountains of data, it is a very iterative process to slice and dice through summaries of data and without the right Business Intelligence technologies, It can be a bear!
Believe me, in our own company's solutions, as you add additional dimensions to a multi-dimensional cube, every slice and dice operation takes more and more time, just because of the computations that are needed to make these analyses happen across many dimensions.
For example, Region-wise and then, Product-Wise breakup of the sales numbers of a product need to be recomputed on the fly, if you were to ask for Product-Wise, and then Region-wise breakup!
I had the chance to meet Brian Gentile, CEO of Jaspersoft, a Commercial Open Source Business Intelligence vendor. He mentioned something interesting that would solve this above problem with slicing and dicing multi-dimensional data at will.
In-Memory Analysis!
This video shows some typical uses of In-memory analysis that may benefit from large volumes of data and more importantly, analysis being led by insights gained by some preliminary analysis.
While looking at the salary data of managers, they notice that some managers are getting significantly less than others. This leads to the analysis of whether gender determines lower salaries in this instance!
This is how most Business Intelligence users gain insights into data. You ask them, what insights or information did they need from the mountain of data at hand, they would not have a clue.
However, you put some interesting tools in their hands and let them play around with the data, insights lead to further insights and all this will happen if the results show up in a reasonable amount of time on the screen.
Usually the threshold is experimentally pegged at around 10 seconds or so for website loading and possibly about 30 seconds if they know that the computer system needs to crunch through data to give them a screenful of information. If it doesnot happen within that time, users lose interest and not gain from some valuable they could have gained!
It is funny sometimes even with extremely fast processors these days, how whether you get the information you need really depends upon some technology advance like in-memory analysis.
Where is all the knowledge we lost with information? T.S.Eliot
.
In memory analysis is a great way to query large amounts of data, however if its not columnar base in memory it hides a big problem from the user.
1. You need a lot of memory to store one data model and in many cases you have more then one
2. In a multi-user environment in-memory solution will cause the machine to return results after a long time due to the fact that the memory is filled with a lot of useless data (Dimensions and measures people are not using)