Sure, the analysts do a good job in describing the technology that’s described to them by the vendors, perhaps a few customer experiences, but I’m not sure that’s the best way to understand what the technology really is, and where it fits into a SOA stack, as well as a typical problem domain and solution pattern. That takes a lab, a lot of time, a lot of money, and some good independent evaluators. But, I think you could make a nice business out of that.

It was during my tenure as a contributor to PC Magazine that I understood the value of such detailed evaluations. We spend days on the criteria, weeks on testing, and really tried to nail the reviews, providing a balanced view of the product set, even selecting a winner. Of course, sophisticated and high priced SOA technology are not development tools, word processors, or printers, but they should be put under the same sort of rigorous testing, including ease of installation, set up, performance, scalability, and maintenance.

Indeed, you may be surprised how some of the leading solutions fall down in certain areas, even vendors that have been in the business for years. This approach gets beyond the marketing, and lays out the real capabilities that you get for your million or so dollars, considering cost of technology and services required.

I’m unable to do this, but somebody should. I think the end users would pay big to get the evaluation work done for them, and get beyond the hype down to the real capabilities of this technology set. Vendors, back me up here.