Wittgenstein's Ruler and business metrics
The more free parameters, the less you know what is being measured. Hence, always couple metrics cannot be relied upon with direct qualitative observations
2023-03-09 by Luca Dellanna
“Unless you have confidence in the ruler’s reliability, if you use a ruler to measure a table you may also be using the table to measure the ruler.”
This heuristic is called “Wittgenstein’s Ruler.” I like to formulate it more precisely as follows:
Principle
The more free parameters, the less you know what is being measured.
Some examples of Wittgenstein’s Ruler
-
When prizes (such as a Nobel) are awarded too often to a non-deserving candidate, awards tell us more about the awarding committee than about the winner.
-
Feedback sometimes tells us more about the person giving it than about its object.
-
When a ranking based on objective metrics is published, the choice of the metrics used sometimes tells us more about the committee choosing them than about what the ranking is supposed to evaluate.
How does Wittgenstein’s Ruler apply to businesses?
Here are a few examples:
-
When you survey your customers, make sure that you choose the questions well so that the results will give you information about your customer’s preferences for your product and not about your own preference for survey questions.
-
When you ask for feedback, use very specific questions and have a track record of responding well to feedback. Otherwise, the feedback you receive won’t be about what it’s supposed to describe but about your relationship with the feedback provider and your mutual expectations.
-
When you choose which business metrics to measure, make sure you pick metrics that describe well what’s going on and don’t instead measure the amount of “gaming” your people do to achieve them or the measurements that you’re comfortably making.
Personally, I believe that in many cases, it's impossible to avoid Wittgenstein's Ruler effect entirely. Instead, it is possible to mitigate it by increasing direct qualitative observation. For example, if now and then you go to your teams' workstations and observe how they work and how metrics are collected, you might get a better overview of what's really going on in your business than any metric might tell you.
I'm not saying that metrics are bad. They're good.
Instead, I'm saying that metrics cannot be relied upon and must always be coupled with direct qualitative observations.
