We use cookies and other similar technologies (Cookies) to enhance your experience and to provide you with relevant content and ads. By using our website, you are agreeing to the use of Cookies. You can change your settings at any time. Cookie Policy.

The Connected Web

Phil Wainewright

Analyst Criteria Skewed Against SaaS

Vote 0 Votes

The way we think about computing is often so ingrained in a particular pattern that we don't even notice how biased we are against newly emerging technologies.

There's a built-in bias, for example, in most cost-of-ownership comparisons between conventional licensed software and pay-as-you-go SaaS alternatives. People typically add up the subscription cost over three years and compare it to the cost of buying and running a conventional software package for the same timespan. What that doesn't take into account is the huge cost and disruption of upgrading to the next version of the conventional software at the end of the three years — whereas the SaaS package, having been continuously upgraded throughout the subscription period, is already fully up-to-date with the latest functionality at no extra cost.

Another hidden bias came to light recently as a result of research company Forrester adding SaaS collaboration vendor Central Desktop to its August 2009 Forrester Wave for Collaboration Platforms. This report compares vendors on 66 separate criteria, but CloudAve's Ben Kepes noticed that at least one of the criteria automatically discriminates against SaaS vendors:

"One of the criteria to assess current offering strength is that the product has cross platform support — companies are given a ranking from 1 to 5. One could be excused for expecting that Central Desktop, being a web based application, would score the highest marks in this area — unfortunately they didn't, because their application cannot be installed on a plethora of systems. What Forrester is actually measuring is cross OS support but their methodology is hugely skewed in favor of the on-premises vendors."

SaaS vendors, of course, spare their customers the burden of installing the software at all, and therefore it's a total absurdity to mark them down for not being available on a range of different operating systems. But that's precisely what Forrester has done.

The story is worth bearing in mind next time someone tells you that a SaaS or cloud option is clearly inferior. Ask yourself what criteria they're judging it on before relying on their verdict.

PS [added Sep 18th]: A reader points out that the publishing timetable and two-year lifetime of analyst product comparison reports is itself a built-in bias against SaaS vendors. It's designed to jibe with the on-premise product delivery cycle, where a new version is designed to last at least a couple of years and is made available for analyst review long before customers have deployed it. Whereas for a SaaS vendor's product, the report is already a version or two behind by the time it comes out, and there will be as many as eight version updates during the lifetime of the report.

"In that same timeframe, the on-premise players may or may not have even delivered one update," he writes. "I would wager only a single-digit percentage of their customers will have deployed that new version. All of our customers are on our latest update."

So it seems it's not just the criteria that are skewed against SaaS — the entire process has an inherent bias. Maybe it's time the analysts themselves switched to a more connected, on-demand model?


A good tco analysis should have line items for software, hardware & resources, which would include your upgrade issues above. I like both models, at the end of the day, it's as much about what's the right fit as cost. The lust of these considerations is 50 items long, which will be part of a proper comparison.

As usual, Phil, this is a very insightful and important piece. Thanks for helping people understand better that SaaS is not just a delivery model, it changes the entire experience and relationship a customer has with a vendor--for the better.

Excellent points all. Mr. Low got it right: "SaaS is not just a delivery model." It is a new business philosophy for how companies buy and view software. Unlike traditional installed software, which are measured on the value of the deal at signing, SaaS is based on the value of the relationship. Because of the pay-as-you-go nature of SaaS, the vendor is incented to align (and adjust) its goals and service with the long-term needs of the customer.

See what I mean here: http://www.supplyexcellence.com/blog/2007/03/06/saas-disrupts-old-school-thinking/

Thank you so much for your very insightful piece. As always, you have articulated the benefits of SaaS very well. A model which uses "platforms" as a uniform criteria to compare on premise and SaaS solutions certainly needs to be updated.

Phil Wainewright blogs about how businesses are using the Web to get better plugged into today's fast-moving, digital economy.

Phil Wainewright

Phil Wainewright specializes in on-demand services View more

Recently Commented On

Recent Webinars


    Monthly Archives