The Danger of Placebo Metrics

When I was fairly young, my father taught me a lesson that has proven valuable in business to this day. Although I was years away from driving age, I started to show an initial curiosity about cars. Recognizing my new interest, my dad took me with him to the local garage to pick up the family car after a tune-up.



I remember being shocked by the $25 bill for the service. [On my meager 25-cents-a-week allowance, this seemed like an expenditure that would require a lifetime of saving to be able to afford.] When I commented on the “mammoth” tab, my dad pointed out that the tune-up was essentially free because the money he would save from improved gas mileage would more than cover the cost of the service.

He explained that before the service, we were getting about 16 miles per gallon, and that it typically improved to 21 miles per gallon after a tune-up. Gasoline was selling at the exorbitant price of 29 cents per gallon in those days. [Yes, there really was a time when you could buy gasoline for only 29 cents a gallon; and yes, I’m old enough to remember it; and no, it wasn’t as far back as the old hand-cranked models, either!] My father went on to explain that given the number of miles we would likely drive before the next tune-up, we would save much more than the $25 we had just spent for the tune-up.

In those days, we drove the car about 10,000 miles a year. At 21 miles to the gallon, that required roughly 476 gallons of gas. At 16 miles per gallon, it would have required 625 gallons. The difference (149 gallons) would cost $43.21 at 29 cents a gallon. This was clearly more than the $25 he had just paid for the tune-up.

This marked the beginning of my understanding of several important concepts. First, I was going to have to earn a lot more than 25 cents a week if I was ever going to own my own car—even if tune-ups did eventually pay for themselves! Second, if you’re going to understand the impact of a change, you’ve got to take the appropriate measurements before and after the change. And third, if you measure the wrong things—even if you do it before and after the change—you won’t know the actual impact of the change.

These last two realizations are even more important today—now that organizations are pinning such high hopes on the likelihood of various forms of integration, automation, and collaboration producing significant business improvements. More than ever, when attempting to determine the business impact of one information technology initiative or another, there comes a point when organizations need to decide what they should measure to track success. All too often, those trying to calculate the cost/benefit of their process improvement initiative latch onto some statistic that measures something, but not something that matters…hence the name “placebo metrics.”

EXAMPLE #1

Imagine, for example, that we work for a company that wanted to avoid expanding its accounts payable staff in response to increasing workload. Our goal would be to minimize the amount of manual intervention necessary to process an inbound invoice. Suppose our approach was to automate as many manual tasks as possible, while also removing error-generating steps (which, in turn, force additional manual intervention).

How could we accurately measure the effectiveness of our efforts? Should we measure how many of our vendors we could convert to automated processing? Or, should we measure the dollar volume of the invoices processed using the new automated approach? In this case, the answer is neither.

Remember that our stated goal is to reduce resource inefficiency by removing work from our manual resources. Therefore, the worth of any integration and automation efforts we might initiate should be measured in terms of how they affect workload. For example, if we chose “number of business partners implemented” as our metric, we would have a placebo metric. We could implement 10 companies, each of which sends us one invoice a year, and one company that sends us 10,000 invoices a year. If we tried to track our success based on the number of partners implemented, the ten single-invoice companies would falsely indicate greater success than our one high-volume partner.

Similarly, we could choose to track the “dollar value of invoices converted” as our measurement. Using this placebo metric, we would count one invoice for $1M as signaling greater progress than 100 invoices for $50 each. The key question becomes “Is the change that we’re trying to affect best measured by the metric we’ve chosen to track?” In our example, we’re trying to reduce workload for manual resources. So, what is the primary driver of work?

The answer is the number of invoices converted to automated processing. Regardless of the number of business partners whose inbound invoices we integrate and automate—and in spite of the dollar volume of invoices processed electronically—the most direct measure of work impact is the generator of the work (i.e., the number of invoices processed).

Certainly, for those companies that process invoices at the line item level, the real driver of work is the number of line items. Unfortunately, few companies maintain detailed records showing the number of line items received on a per vendor basis. Therefore, the most accessible and, yet, appropriate measure is the number of documents converted from manual to automated.

EXAMPLE #2

Imagine that we wanted to reduce our inventory levels and thereby, cut inventory carrying costs. Suppose that we had determined that our receiving process for handling inbound shipments and getting those products into inventory was painfully slow. We realized that these delays in our receiving process were causing us to carry additional inventory to service inbound orders (because our order processing system could not allocate inventory to inbound orders until the physical product had been received and entered into our inventory management system).

In this example, the number of business partners converted to the improved receiving method would be a placebo metric. The change we’re trying to reflect is the dollar value of inventory, and the number of partners converted does not necessarily have a direct corollary to inventory carrying costs.

Similarly, we would be remiss if we measured the number of shipments received via the new method. Converting a great number of low dollar value shipments would not have as dramatic an effect on inventory carrying costs as a handful of high value shipments. [Note: While it is true that handling more shipments certainly takes more work, the impact that we’re trying to measure is on inventory carrying costs, not work levels.]

Because we’re trying to reduce the dollar value of inventory, the greater the dollar value of the shipments we convert to the improved receiving method, the greater the impact on our intended target.

EXAMPLE #3

Finally, suppose that we wanted to synchronize our product information with our suppliers (i.e., establish and maintain consistent, accurate information between ourselves and our suppliers). Is there the potential to fall victim to placebo metrics in data synchronization? Yes. Anything that has the potential to improve a situation has the potential for its success to be incorrectly measured. So, then, where are the potential placebo metrics in data synchronization?

Consider, with data synchronization, the goal is for the source of the information (typically the manufacturer) and the recipient of the information (us) to both have consistent, accurate information within and between our two systems. A key step in this process is for the source to electronically publish the information to be synchronized to us in a timely manner.

Since the electronic publishing of the information is a necessary—and highly visible—step, many recipients are prone to mistakenly measure their success with data synchronization based on (1) the number of sources publishing to them electronically and/or (2) the number of SKUs (stock keeping units, a.k.a. items) being published to them electronically. While these measurements certainly indicate progress in avoiding the need to manually key-enter this product information at the recipient, they are a placebo metric when it comes to measuring the success of the overall data synchronization project.

Again, consider the end result we’re trying to produce with data synchronization…consistent, accurate information within and between ourselves and our suppliers. What changes when we are successful at this? The answer is threefold.

  1. Because the information is now being received electronically by us, it need not be key-entered. This means that there is less manual intervention which allows us to liberate staff who can be applied to other tasks.
  2. Because the item information is received electronically, it gets into our systems faster. This allows us to get new items to market faster.
  3. Because the information is consistent and accurate within and between us and our suppliers, there are fewer exceptions in our daily transacting of business (i.e., purchase orders, shipping notices, invoices, etc. are more accurate).

None of this can be measured by the number of partners publishing or the number of SKUs being published electronically. Rather, data synchronization success must be measured in the following categories:

  1. Reduction in key-entry time (for us, the recipient)
  2. Reduction in time to update systems (for us, the recipient) and, therefore, to get new items to market
  3. Reduction in business exceptions and the resultant manual intervention and operating costs associated with those exceptions (for both us and the source)

Using these, we can accurately gauge the progress we’re making with global data synchronization.

IN SUMMARY

Albert Einstein once said, “Not everything that can be counted counts and not everything that counts can be counted.” As a general rule, we can avoid being a victim to the placebo metric by paying close attention to the end business result we’re trying to achieve. By focusing on our ultimate target, it becomes substantially easier to determine whether the measurement we intend to monitor will, in fact, be a representative barometer of progress.

My father wanted to be able to cost-justify his investment in the tune-up. He certainly knew that it was in the long-term interest of the car’s well being to keep the motor tuned, but he was looking for a short-term payback, as well. He knew that besides ongoing maintenance, gasoline expenditures represented a direct cost of using the car. He knew the amount of gas a car used was related to the car’s performance efficiency, and that a tune-up would affect that efficiency. So, he measured the before and after change in gas utilization performance efficiency—a.k.a., miles per gallon—to best gauge the impact of the tune-up. He could have elected to track the number of hours the car was driven or the number of trips taken, but because these measurements did not affect his target results, he saw them as the placebo metrics they were.

Each of our organizations will continue to attempt to leverage information technology for the betterment of the company. However, we will never know whether or not—and to what degree—we’ve been successful unless we (1) decide to capture the before and after measurements (so we know what changed and by how much) and (2) choose to track the right metrics. If we ask, “What must change to produce the result we’re pursuing?” and “Which measurement—if positively affected—will consistently and accurately signal a positive change in our desired result?”, we can be much more confident that we’re tracking measurements that matter...not placebos that sound good, but mean nothing. After all, wouldn’t you like your electronic commerce investment in time, effort, and money to produce meaningful, measurable, and reportable results that matter rather than just a “To-Do” list with a series of checked-off tasks?

About the Author

John Stelzer is Director of Industry Development for Sterling Commerce. Since 1984, he has been providing education and consulting on electronic commerce—to date, educating more than 27,000 professionals from over 16,000 companies. For more information on electronic commerce in the retail industry or data synchronization specifically, John can be reached at 614.793.7046 or john_stelzer@stercomm.com

More by John Stelzer

About Sterling Commerce

Sterling Commerce is one of the world’s largest providers of business integration solutions. For more than 25 years, thousands of companies have depended on Sterling Commerce expertise to optimize collaborative relationships through the integration of applications, external partners, suppliers and customers. With more than 25,000 customers worldwide, Sterling Commerce is the dominant business integration solutions provider in retail, consumer packaged goods, manufacturing, financial services and telecommunications.