Integration on the Edge: Data Explosion & Next-Gen Integration

I am highly critical of those who promote crowdsourcing as some sort of panacea. Except for Duct Tape, nothing is a panacea.

Crowdsourcing has many of the same shortcomings as outsourcing, and a few more to boot. Some issues I see:

1) For the most part, the really talented architects and developers are busy with lucrative and demanding jobs. So if you think you're going to get someone to create something better than Hadoop without any compensation guarantees, think again. I'm not saying that there aren't some highly skilled people out there. But one needs to be realistic. You can't open something up to crowdsourcing and expect the world to come to you. On the other hand, your particular problem might be perfect for someone out there.

2) Although "contests", such as those promoted by organizations like TopCoder, can be useful for improving the quality of potential solutions, the quality of the end-solution is mostly determined by the quality of the talent pool and how well the contest is set up and managed. The fact that there is a contest is incremental only. And don't forget the overhead of managing and "judging" any such competition. It's not as easy as determining the winner in a 100 metre dash. As always, a badly defined and poorly run competition is almost certainly a failed one.

3) It's human nature that results and motivation go hand in hand. In the working world, motivation takes many forms, and every person is different. In general, people are motivated by compensation (monetary and non-monetary), respect and visibility from others, a sense of being part of a team, and the satisfaction of a job well done. Because the nature of the relationship between a crowdsourcing participant and the business is so very ephemeral, the typical person simply doesn't have the underlying motivation.

4) The mathematics of complex problems are against outsourcing in general. By this I mean that the effort to successfully manage such a project increases geometrically relative to the complexity of the problem trying to be solved. Specifically, the effort to fully document the requirements of a project, to quality assure the results, to ensure that the proposed solution not only meets the requirements as set forth, and provide a sound and extensible architectural base for the future increases MUCH faster than the actual effort required to solve the problem.

If you get involved in trying to manage multiple disconnected or semi-connected parties with varying competing or complementary solutions, that makes things even worse.

5) Who's going to maintain all that code? Years ago, I had to write a utility that extracted data and metadata from a proprietary column-store database and stuff it into Oracle 10. A year after I wrote the utility, I needed to go back and modify it. It's difficult to maintain your own code. Maintaining someone else's code is far more difficult. Although turnover is common in any organization, supportability of in-house developed code is significantly easier than externally acquired code.

6) Collaboration with internal team members is an important part of most development efforts, and this is something that is difficult with a crowdsourcing solution.

All that said, crowdsourcing has some real benefits. True advantages. For some projects (or project components), it's a great idea.

1) It can be highly cost effective when used properly (i.e. managed properly and employed for an appropriate class of problem).

2) The wide range of potential problem-solvers out there mean that some very interesting and innovative solutions can pop up.

3) External people aren't constrained by the mental "boxes" and preconceived notions that internal people create for themselves.

4) When used properly, the downside of crowdsourcing is pretty minimal.

Anyhow, all this comes down to the old maxim: use the right tool for the job. Some projects just aren't cut out for crowdsourcing, and some are.