Most organizations getting data integration working are so happy that it is indeed working, that they often do not consider performance. The result is dysfunctional data integration latency, or information not appearing in target systems as needed by the business processes.
There are a few problems to address here. First, are data integration performance issues that are caused by unplanned organic growth. Second, are data integration performance issues caused by bad data integration architecture. Let's address each.
Organic growth of the source and/or target systems or databases causes performance problems when the data integration system is tasked with searching through the data for new or changed records. Thus, when there is only 10,000 data records performance and not an issue, but at 1,000,000 records, searches could take several minutes.
The result is that the data integration system can't gather the information needed in time, and thus it's not delivered in time. This can occur in both real time data integration and systems that use a batch model. The end results are the same, information not being where it needs to be.
The most common issue, however, is bad data integration architecture. Typically this means not understanding the core data integration requirements around performance, and thus selecting the wrong approach and technology. In most cases I've found that this is really around leading with technology versus understanding the business and architectural requirements, around performance but also data integration in general.
Examples of this include leveraging "real time" data integration when batch is indicated. Or, not understanding the real world issues around data movement, including network and system saturation. Or, most common, forcing a square technology peg into a requirements round hole. For example, talking about using a traditional EAI solution when nothing is understood about the problem domain as of yet.
With a bit of planning, performance problems around data integration are avoidable.
Leave a comment