To be competitive, and to deliver the services that its customers expect, an enterprise today must integrate information from many different data stores and applications. This is difficult, because each information source has its own peculiar data structures and formats, and its information must be transformed before it can be understood at its destination. As application integration expert David Linthicum puts it, the transformation layer is the Rosetta stone of the system: it understands the format of all information being transmitted among the applications and translates that information on the fly, restructuring data from one message so that it makes sense to the receiving application or applications.

The need for such a layer is emphasized by the move to Service-Oriented Architecture (SOA), which requires exchange of information between loosely-coupled services. But implementing it is costly. Data transformation can take up to 40% of an organization's typical programming budget, and this is a lot of money for a process that simply rearranges information without adding intrinsic value.

For example, a manufacturing company may need to supply product information to its customers, in different customer-dictated formats. That information may be derived, not only from its own applications and databases, but also from those of its subcontractors. The end customer looks for integration of the supply chain, but this can only be delivered through transformation of the information that flows between the links. The cost of providing that transformation cuts into the profit margin, and the CIO is under constant pressure to reduce it. What can that CIO do?

Automate the Transform Development Process
The logical answer is to improve the productivity of the people programming the transforms or, ideally, to automate the transform development process. It is straightforward and repetitive in nature, seemingly a good candidate for automation. However, it requires an understanding of the meaning of the information being transformed, and this is a degree of machine intelligence that has not yet been achieved.
Back in 1950, mathematician Alan Turing gave a definition of machine intelligence that is still accepted today. He said that a machine should be regarded as truly intelligent if a person communicating with it could not tell, without seeing it, whether it was a machine or another person. With that level of intelligence, a machine could understand descriptions of data produced by people sufficiently well that it could transform the data. Or it might even develop the descriptions itself, by understanding the data.