We use cookies and other similar technologies (Cookies) to enhance your experience and to provide you with relevant content and ads. By using our website, you are agreeing to the use of Cookies. You can change your settings at any time. Cookie Policy.

Integration on the Edge: Data Explosion & Next-Gen Integration

Hollis Tibbetts

Building Integration Yourself - Possibly the Dumbest Idea You've Had in a Long Time

Vote 1 Vote

The more things change, the more they stay the same

I remember, back in the mid to late 1980s, talking to IT departments about the concept of a relational database and why having one on their VAX would be a good idea.

In many cases, my recommendation was met with some head-scratching, brow-furrowing and comments along these lines: "Why would be buy one of those, when we can just build our own data storage system using RMS records?"

A hundred years later, it's a pretty rare occurrence that someone would decide to build their own data storage and manipulation system; if they do, it's because of some unusual requirement. As far as I know, nobody considers "build" as the default strategy in this area.

Integration: The Debate Still Continues

The build vs. buy debate still rages on when it comes to data or application integration, though. I've lost count of the conversations I've had that include phrases such as "it's just a few web services" or "the API is easy to write to" or "it's just a simple point-to-point integration".

If you want to hire me to evaluate your situation and make a recommendation on whether you should build your own integration "mechanism" or just buy one, that's great. Be forewarned that 994 times out of 1000, the answer will be: "Go buy one".

Oh, and the remaining six times? Five will be "could go either way" (which means "buy"). Only one would be "Build is better for you".

What is the Real Cost of Building Integration?

Here's the No. 1 reason why building your own integration solution is probably a terrible idea: almost everyone dramatically underestimates the cost to the organization of building, QA-ing and maintaining even the most simple of integration mechanisms.

Odds are, whatever you think it's going to cost, the real cost is several times that.

Cost of the Simplest Integration

Let me give you some numbers from a highly respected research and consulting group - they happen to come from Gartner Inc, which has done extensive analysis in this area.

Let's examine what I consider to be one of the simplest possible integrations: a relational database with 12 columns per table on average, and 20 tables as a data source (this is just one-way integration, out of the database). To make things easier, this would be for a small to midsized privately held business. Things get much more complicated in large companies where multiple departments get involved or where there is the need for increased levels of governance. Again, this is the simple and easy scenario.

Factoring in business analysis, technical analysis and design, construction and coding, QA, deployment, we're looking at 234 FTE (full-time equivalent) days of effort. At a fully-burdened rate of $85 an hour, that's almost $160,000.

We're SOA/Web Services Based - It Should Be Simple!

Some of you may be thinking "we're pure SOA" or "we do everything through Web Services", so that would make integration a lot easier to custom code.

When considering the complexity of the intended end-point for integration (i.e., a source or a target), services should never be considered simple in terms of effort to code interfaces to. Data services that wrapper existing application processing in a web service will always require significant examination for abstraction appropriateness, which places them in the "moderate to complex" category in terms of effort required. This raises the effort required to get them to the point of deployment by a factor of 3 or more.

Fine-grained services should always be considered complex targets - plan on a factor of 6x.

Services can be quite complex to interface with. Think about scenarios where an information artifact is decomposed into multiple different services, separating out data and metadata. These services need to be re-composed by the integration interface. Very complex indeed.

So Why is Building Almost Never a Good Idea?

Well, I've tried to establish that even a simple integration project is very expensive to build. Plus, integrations are a bit like mice: You rarely end up with just one of them. You usually have at least two, and before you know it, there are dozens.

The scenario we just discussed just for one-way integration. And that $160,000 is just to get the project deployed. It doesn't include any support, maintenances, changes, updates, etc.

The three-year cost of this simplest of integrations can be far, far more.

The nasty thing about build-your-own integration projects is that they don't increase linearly with complexity and quantity of sources / targets. They are most definitely a geometric progression. Go from two sources to 20 and it's not 10 times as expensive--it's more like 100 times or more.

Is Buying Really Better?

I don't for a moment pretend that integration software costs nothing. You have to buy the software, train people, implement it and support it.

It's reasonable to say that a packaged integration solution will reduce analysis, development, support and enhancement costs across the board by 75% to 90%. The bigger the project, the more change involved, the bigger the percentage savings.

Isn't Integration Software Expensive?

Back in the late 1990s and early 2000s when I worked for a company called webMethods, the average transaction was in excess of $150,000.

I'm not an expert in software pricing for all the different vendors out there, but taking a look at some prominent and well-regarded players gives you at least an idea of what the price-points are:

Dell Boomi: their Professional and "most popular" edition is $1,100/monthly (http://www.boomi.com/products/editions/pricing)

Informatica's Cloud Basic is $1,500/monthly, their "most popular" Standard Edition $3000/monthly. (http://www.informaticacloud.com/products/editions.html)

Mulesoft's iON Plus "most popular" version is $550/monthly (http://www.mulesoft.com/mule-ion-pricing)

SnapLogic's SnapServer costs "upwards of about" $800/monthly, according to an interview in Information Age. Their "Snaps" range in price - many are free, none are expensive. Pricing is available at the Snap Store. (http://store.snaplogic.com)

(note: these vendors are listed in alphabetical order).

These numbers aren't designed to be a definitive price quote by any stretch of the imagination, but they at least give you an idea of what kind of price points we're talking about for basic integration - which is to say, not a lot of money considering how much easier it will make your life.

And not much money when you consider how much faster your project will go into production (which is when your company starts reaping the benefits of the investment), how much easier the products will be to support, and how much more scalable (in terms of performance as well as architecturally) they are.

Still not convinced?

There's just no convincing some people. If that's you, maybe you should just go build your integration system and see how that works for you.

But before you launch yourself into that tar pit, consider the following questions and bits of information:

1) In the SaaS world, APIs are updated on average 4-12 times a year. What is the impact of that on your custom code? What if a document format (e.g, EDI document) changes? Will you even know in advance of these changes, or will the change happen and suddenly your system stops working and you have a crisis on your hands?

2) Are you prepared to handle latency and unavailability issues, timeouts, etc.?

3) Have you budgeted for building a sufficiently robust logging system for errors, as well as for when data ends up somewhere it shouldn't and you need to undo the situation?

4) Can you guarantee that data won't get lost when something "bad" happens?

5) How will you monitor what's going on in the system?

6) Coding transformations and business logic in Java, C#, C++ or any other programming language is very time consuming. Transformations and business logic change a LOT. How will you support that? Most integration products support simple or standard scripting languages, drag and drop, reusable objects, etc.

7) How do you plan to implement mapping - especially between something like a web service and a relational database (where one can be hierarchical in nature and the other a collection of tables). What happens when one of those things changes? Have you thought about transactionality and serializability? Do you need to support that? How will you do it?

8) Many applications require the use of proprietary SDKs for integration. Are you trained in those? Are you prepared to support changes in the SDKs?

9) What levels of performance are required? How do you plan to meet those? What happens if that changes? Is scalability built into your solution?

10) If more sources or targets for integration are added, will your system support that, or did you build something that is a throwaway?

11) Does your home-built system support concurrent development?

No TrackBacks

TrackBack URL: http://www.ebizq.net/MT4/mt-tb.cgi/18220



It's unclear what you mean when you say buy-vs-build here. For example, the first step in the ION Documentation is "Build" and advertises "Be a developer".

Perhaps you are talking about the integration platform. But if so, I think you need to be careful about how you frame this. A lot of management doesn't understand that once they buy these packages, they still need to pay for many tens-of-thousands of hours of development.

Effectively, these are development platforms. The biggest problem with what I've seen with many of them is they start by trying to simplify things down to a level that can't be used to do anything useful.

Any clarification you can provide would be useful.




I have to agree with James here. Although you present a reasonable case that "integration is more difficult than it seems", I don't see any really compelling case for buying rather than building - of the 11 issues you list at the end, 1, 3, 5, 8, 9, 10 & 11 all exist with integration platforms like webmethods. And it's not as if you don't have to do business analysis, technical analysis and design, construction and coding, QA and deployment of the purchased solution. As somebody with quite a bit of experience wading through these integration platforms, I think the biggest issue they face is that they oversell themselves rather than being honest and upfront about what they're really for.


Good piece. The other comment I would make is that more and more application integration is being driven by the business side, and the needs for being agile, and rapid time to value do not match the old enterprise class tools.

I am also building a LinkedIn group to educate and promote "Application Integration"

This blog offers an informed and informative perspective on the ongoing explosion of data and the technologies used to turn this data explosion into assets and competitive advantages.

Hollis Tibbetts

Hollis has established himself as a successful software marketing and technology expert. His various strategy, marketing and technology articles are read nearly 50,000 times a month. He is currently Director for Global Marketing Operations for Dell Software Group. Hollis has developed substantial expertise in middleware, SaaS, Cloud, data management and distributed application technologies, with over 20 years experience in marketing, technical, product management, product marketing and business development roles at leading companies in such as Pervasive, Aruna (acquired by Progress Software), Sybase (now SAP), webMethods (now Software AG), M7 Corporation (acquired by BEA/Oracle), OnDisplay (acquired by Vignette) and KIVA Software (acquired by Netscape). He has established himself as an industry expert, having authored a large number of technology white papers, as well as published media articles and book contributions. Hollis is a top-ranked author on Sys-Con media, is also published on Social Media Today "The World's Best Thinkers on Social Media", and maintains a blog focused on creating great software: Software Marketing 2013. He tweets actively as @SoftwareHollis Additional information is available at HollisTibbetts.com All opinions expressed in the author's articles are his own personal opinions vs. those of his employer.

Recently Commented On

Monthly Archives