We use cookies and other similar technologies (Cookies) to enhance your experience and to provide you with relevant content and ads. By using our website, you are agreeing to the use of Cookies. You can change your settings at any time. Cookie Policy.
Start a Discussion

What's the Best Way to Solve BI's Data Integration Problem?

Vote 1 Vote
In this IT Business Edge blog by Loraine Lawson, Can Data Virtualization Solve the BI Integration ProbleM? Howard Dresner is quoted as saying that data integration is the chief complaint about BI implementations.  So what's the best way to solve the data integration problem?

14 Replies

| Add a Reply
  • Data Integration and humungous ETL efforts are here to stay for good and will always be a problem. The reason is that functional software have all evolved over a period of time starting with individual departments and functions - Financial Accounting, Warehousing, HR, Sales, Manufacturing - all having evolved from departmental and functional focii. Now we want an organizational view, a single view of the customer from each of the different functional views and you will have to do ETL. May be some 25 years from now, we will have Process-based software replacing functional ones and then we can have a single view of entities like customer, order, etc. Then ETL and Data Integration may not be that big a deal of the total effort like it is currently!

  • Of course, one way is to avoid BI data integration problems in the first place - go back 1 level in the architecture and do event-driven integration! Handle the "data" whilst it is still an "event"!

  • Begin with the decision in mind - don't integrate anything that does not help you improve the decision you are focused on (you are focused on improving some decision, right?). Check out this post for some thoughts.

  • Though there is a dependency, BI and Integration should be dealt with separately as one is business driven and the other is more to do with IT.

    What could possibly bridge the gap are EDA and SOA from an IT perspective coupled with a business focused framework which allows business users to easily access information – historical or real-time.

    It’s all about delivering information to decision makers when they need it the most!

  • Data integration and ETL are aiming at ever changing targets, both sources (transactional systems) and destinations (KPIs, measurements, signals, alerts, etc.).
    Let me think about it from a slightly different angle. It may not be the best way, but it might help you, I hope.
    Do you really need that precise information to make decisions? Conceptually, say, most management people want to have 80% accurate information in the last week, than having 99.9% accurate information tomorrow. “Know enough? might help you, even if it’s not perfect or the best.

  • Two answers directly from the experience of many of our business intelligence customers . . .

    1. For well-structured data environments where the source systems are not changing frenetically and are defined well, using a more traditional ETL -> DW approach can be cost-effective (especially nowadays), feature-ful, and high-performance.

    2. For less-structured and more far-flung data environments where change is the norm, using virtual views and federated queries (a la SOA and EII) can be flexible and extremely powerful. In fact, we have some large customers who swear that this is the most effective way to query data and bring the most timely information to a decision point.

    Having a variety of options that span an even wider range than these two describe is a major benefit of current generation BI technologies. It's good to have technological choice that can be matched up against any given business problem.

    Brian Gentile
    Chief Executive Officer

  • A significant part of the problem with BI and data integration is that we are individually the biggest data integrators. Combine that with the silos of vital business data sitting in places like spreadsheets and you will always have a fragmented picture of what the organization knows, limiting the ultimate potential of business intelligence and analytics in general.

    We must make business data more open, transparent, and accessible. Providing quality IT and business support for mashups, self-service BI, business data feeds from all apps, social and otherwise, will go a long ways towards addressing the issues we see in this space.


    Dion Hinchcliffe

  • So far, all of the comments above provide definite insights into handling the issues surrounding BI and data integration issues, especially because companies will differ in their business problems and their approaches to addressing data issues.

    I think handling data issues within originating source systems is the ideal, but unfortunately rarely the reality. Many companies argue between IT and business unit control over data within source systems based on the processes surrounding the information being argued about. Consequently, data integration, and more importantly the processes surrounding the development of cleansed data with added data quality initiatives becomes essential when looking at how to solve integration issues. I don't think there is one sure way to deal with data integration problems, but do think that they are impossible to overlook and that if organizations do not develop a process oriented approach to managing their data, the ability to develop a successful BI infrastructure will be elusive at best.

  • The core problem is that most enterprise integration and BI is centralized, while the data is distributed.

    The solution is to move to a more distributed integration model, where data is readily available as a service to the individual. The individual is already performing much of the innovation, exploration and integration, and this distributed model facilitates the process.

    Once the early innovation stabilizes, it can be transferred to a more centralized process, which can add additional value through improved quality, stewardship, resuability, and reliability.

    Much of the innvoation we see on the web has been driven by this very principle of distributed access. As the number of relevant external data sources expand, a distributed approach may eventually become the only viable one.

  • The success of a data integration project for business intelligence efforts will depend on many factors above and beyond the capability of the integration solution and the project team. Here are ten steps for improving the success of your next data integration project.

    1. Establish a Center of Excellence. Considering the fact that data integration solutions will likely be used in many different areas of a business, it is critical for the long term success of any project to establish a center of excellence that will focus on such things as ensuring a set of standard best practices, strong education on the data integration tool, a common methodology and a set of core standards for pursuing business intelligence projects that involve data integration.

    2. Reusable Services. Most processes created in a modern data integration solution will support modularity and reusability. However, these tools are also able to expose the data integration process as a service in the form of a Web service, enterprise JavaBean or Java Message Service (JMS) interface. When reusable data integration services are leveraged in new data integration projects, this adds to future ROI for the project.

    3. Continuous Training. No matter how expert you feel you are in a particular data integration product or business intelligence solution, it is important for you and your colleagues to continue to expand your expertise by learning as much as you can about the functionality of the data integration product. This continuous training strategy can also serve as a means for expanding the expertise of the internal team so that over time, you become less reliant on third party consulting for basic work .

    4. Don't Get Lazy. All organizations are under a great deal of pressure these days to deliver results from IT efforts. Don’t let pressure undermine the true potential for success in your BI and Data Integration efforts. As you are building a data integration process, keep in the back of your mind common questions that most great programmers ask themselves every day; Is there a more efficient way of doing this work? Can I squeeze as much out of this functionality as possible without adding more overhead? Focusing on building best in class data integration processes by leveraging excellent training, mentoring and expert colleagues is a good way to avoid pitfalls during a BI/data integration project.

    5. Assign, Grow and Nurture a data integration architect. You could probably go out and recruit and hire a strong technical architect with great experience in data integration, however, a better approach is to grow and nurture an internal architect who has the passion, reputation, respect, community spirit, vision and business knowledge and experience to guide colleagues in their approach to data integration. This data integration architect should also be a core member of the Data Integration Center of Excellence and provided incentive to ensure success of each and every BI/data integration project.

    6. Establish an Internal BI/Data Integration User Group and Meet Regularly. It is strongly suggested that you establish an internal data integration user group to ensure that all technologists have a forum for sharing experiences, tips, tricks and insight on the use of the BI and data integration solution. It is also a good forum for inviting your data integration vendor to bring in experts with advice and advanced support for using the data integration tool. Lastly, it makes plenty of sense to send a number of data integration experts to the software vendor’s annual user conference. This is a place to gain education on the existing and future versions of the data integration solution.

    7. Establish an Internal BI/Data Integration FAQ and Publish It on Your Intranet. As your BI/Data Integration expertise grows, you will learn lots of tips, tricks and advanced techniques. Developing a frequently asked questions document to help newly minted Data Integration developers will serve to streamline the learning process and help rapidly answer common questions.

    8. Secure a Vendor Executive Sponsor. Since data integration is very strategic to most every organization, especially in the context of business intelligence, it is important to have an executive sponsor from your data integration software vendor assigned to your account. This executive sponsor won’t replace important support, services and sales contacts for the vendor, but can serve as the business partner for your company. You can establish a long term relationship with this executive who can share your passion about success in using their products. They can help facilitate strategic discussions as it relates to the use of the products and should be accessible for any immediate assistance that you may need.

    9. View Meta Data as a Strategic Component of Your BI/Data Integration Project. Today’s enterprises are facing major challenges when it comes to responding to the significant amount of data being generated and handled by their organization on a daily basis. The vast majority of this data will also be required in the future in order to respond to customer inquiries about how their data is being used, how their data has been exposed to other companies and individuals, and who has access to or has touched the data.

    The healthcare industry is facing this challenge today driven by HIPAA regulations that require health care providers to respond to customer inquiries about the use of their data. Executive inquiries like “where did you get that data, or “I don’t believe those numbers?, combined with exponential growth of data point to an ongoing focus on the critical importance of tracking and managing meta data in the enterprise.

    10. Injecting Evangelism, Passion, Excitement and Focus: Firing Up Your BI/Data Integration Team. Winning teams today are more fired up about their products and solutions than ever before. In the data integration, there are lots of great technology evangelists that are getting highly trained and educated on how to execute BI/Data Integration initiatives and projects to help their companies become highly successful on projects such as providing a single view of enterprise data and customer relationship management. Their success creates great excitement and enthusiasm in their organization for ensuring their enterprises are leveraging the gold that exists in all their data. Having energy and passion around this topic on an BI/data integration project will go a long way.

    With these simple steps, most any organization can turn their next data integration project from just another mandatory IT task into a strategic benefit for their business. By enabling your organization to leverage the value of your data on demand and expand that across the business, data integration holds the key to new responsiveness to business and competitive needs, and resiliency to handle most any situation.

    Bob Zurek,
    CTO and VP, Product Management

  • We all know that any BI effort stands to be confined to the scrapheap unless it is unpinned by a solid and robust data management foundation. To be successful in any BI initiative users need to have a high level of trust in the data they are using so that they can make confident fact-based decisions, rather than relying on guesswork or a ‘gut feel’.

    However, reaching this level of confidence is far from an easy task. Most industry estimates predict that around 80% of the IT effort involved in building a BI system revolves around sorting out the back-end data integration and plumbing. From a technology perspective enterprises need to blend a range of technologies including data integration, data quality, metadata and master data management for integrating, reconciling and cleansing their information.

    Importantly however – and despite what some vendors may claim - data integration has more to do with data governance than just pure technology. Any enterprise data integration effort also needs to wrapped up part of a more formalised data governance effort that provides a working framework for orchestrating the policies and controls for managing and delivering trustworthy data. A successful data governance effort ultimately can make the difference between success or failure for BI and help an enterprise realise the true value from their corporate data.

  • This is a loaded question! I am sure there are as many answers as there are readers. In other words, there is no one "right solution" that will suit everyone. The BI & Data Integration need of a 10 person organization will be dramatically different than a 100 person organization or a 1000, 5000, 10000 person organization. The larger the organization, the more complex the people, processes, systems and data within the organization! Having established that fact, we can look at the basics of BI & data integration.
    1) BI Need: Thoroughly understanding the BI needs of the organization is key in trying to come up with a practical solution. Sometimes simple solutions are the best solutions. Define if the BI needs are focused within department, division, organization or the enterprise. Find out what the BI is used for - is it for measuring the current, comparing the current with the past or predicting the future?
    2) Data Need: If the BI needs are clearly understood, it will be a big step towards understanding the data needs or the data integration needs. For any meaningful BI, the data must be Consistent, Comprehensive, Current and Clean.

    If the BI need is limited to the department, the data integration problem is comparatively easier to handle compared to if it is organization wide or enterprise wide.

    There are plenty of solutions to handle data integration - from simple to highly complex. Some of the technologies have been around for 2-3 decades while others are fairly new and unproven.

    The best solution is never only technical. It is always a combination of good process, good technology, consistent application of processes and technology, and finally, continuous improvement.

    Organizations have seen more success when they take measured steps towards an ultimate goal. The key ingredients are having a core group of dedicated people, consistently applying proven techniques towards reaching the goal.

    Eash Iyer, Vice President, Tychio, Inc. (www.tychio.com)

  • Traditional data integration methods like ETL are suited for bulk data integration tasks that require unparalleled performance. On the other hand, technologies like data virtualization are better suited for use-cases where rapid delivery of on-demand data in crucial. Read more about using data virtualization for data integration here.

  • It's finding the right combination of integration tools that require the least amount of work to tie them together. But if you can find one solution that includes as many different integration tools as possible, the better. there are more tools available today to make the lift much easier that enable enterprises to use their data faster and in new ways. This is just one platform, that my team and I are begging to use https://www.bisok.com/grooper-data-integration-platform/

Add a Reply

Recently Commented On

Monthly Archives