Integration on the Edge: Data Explosion & Next-Gen Integration

Even On-Premises Applications can Leverage the Cloud by Using Cloud-based Data Services

Note: This is the fourth article in a 6-part series on why business software vendors should consider data and application integration an imperative, and an overview of how ISV's can quickly benefit from leveraging integration platforms.

Although part of a series for business software vendors, this article, is just as applicable for anyone with existing on-premises applications who are interested in leveraging the power, scalability and elasticity of the Cloud without having to necessarily re-architect the application or re-host the application on a Cloud.

What we're talking about here is "Cloud-based Data Services". Cloud-based data services is essentially leveraging integration software (or software with integration capabilities) to send requests and data from an application (which can be on-premises or SaaS/Cloud-based) to the Cloud where the request and data are processed, possibly in conjunction with Cloud-based, SaaS application-based or on-premises data.

How is this different from Data Virtualization?

The difficult thing about definitions is that there are so many of them, and people tend not to agree on any particular one. I'm not trying to create or establish any definitions - I simply want to point out a good use case for Integration technology. And if I can start out with a term that some people are familiar with, then great.

Many people are familiar with the concept of data virtualization.

It is very similar (and some might argue the same as) Cloud-based Data Services. I'll simply lay out what I see are the similarities or differences.

Data virtualization is the process of abstracting, transforming, federating and delivering data contained within a variety of information sources so that they may be accessed by consuming application or users when requested without regard to their physical storage or heterogeneous structure.

Data virtualization hides the technical aspects of stored data, such as location, storage structure, API, access language, storage technology, etc.

Cloud-based Data Services do all of the above. In addition to that, they (by definition) reside on a Cloud platform.

Cloud-based Data Services also re typically much more compute-intensive or processing intensive than Data Virtualization. Rather than just bringing data together and doing run-of-the-mill transformations, the elasticity and scalability of the Cloud are leveraged to do highly intensive work - such as simulations, statistical computations, modeling, compute-intensive algorithms etc.

Finally, many people equate "data virtualization" with the term "EII", which carried with it an implication of "real-time" fetching of data. Cloud-based Data Services carries with it no such implication.

Data Services Examples

The attached figure would be a depiction of a scenario where a user running an application (which could be a SaaS application or an on-premises application) initiates a request to a Data Service. That request would typically be accompanied by data.

The request is then fulfilled by the Data Service which pulls data from the appropriate places (on-premises data or applications, SaaS Applications, Cloud-based data), processes the data, "crunches it" (i.e. runs it through some compute-intensive mechanism) and then usually somewhere along the way (but not necessarily) sends something back to the originating application.

One company I spoke with some time ago uses data services to leverage Cloud scalability and elasticity in conjunction with an on-premises application that they build and market. They wanted to extend the application, but wanted the new functionality to reside "in the sky".

The application uses a highly data and compute intensive modeling algorithm - it uses a Data Service deployed on Amazon's EC2 to model phenomena with significant uncertainty in inputs and many coupled degrees of freedom. It relies on repeated random sampling of large amounts of data to make predictions about outcomes.

Hybrid Applications

This particular use of integration technology has enabled a hybrid application - where portions of the application run on-premises and other parts run on the Cloud. Although some work was necessary in the application to enable this, a re-architecture was not required.

Although the example I'm referring to was a traditional "land-based" on-prem application, it could have just as easily been a SaaS application, or a mobile application.

Extending applications into the CLoud is yet another great use of Integration technology.