Using statistics: The case of a drunken man and a lamppost
Andrew Lang was a Scottish poet, novelist, and literary critic, and contributor to anthropology. He now is best known as the collector of folk and fairy tales, including the works of Brothers Grimm, Charles Perrault and Hans Christian Andersen.
But I know him for a very quotable quote “ He uses statistics as a drunken man uses lamp posts — for support rather than illumination.”
While Andrew Lang described the above behaviour in relation to how politicians use numbers that are convenient for them and not really to educate anyone, there is some point to it when it comes to data analysis.
Today, using technology, we are able to measure everything — from every small private details of our lives and health to the world of business where we have hundreds of dashboards with dials not less than those on a spaceship.
But as we make sense of these numbers we need to put some thought into how we use them.
- We need to ensure that we don’t miss the woods for the trees. We should always keep an eye on the larger picture and our ultimate goal.
- We need to ensure our tools are tuned correctly. Errors in measurement can result in a garbage-in-garbage-out decision making loop.
- We need to plan the information we want to collect very early so that we don’t have a lot of missing data points.
- We should look for biases we may have which could flaw our conclusions.
- Correlation does not mean causation. We will need to be scientific about measuring root causes.
Numbers are emotionless and can be trusted blindly but only if the above is true.
So maybe the way a drunk person uses a lamp-post to support themselves is not all that wrong- numbers alone are not enough- we need to know what to do with them. And numbers alone are not going to “illuminate” anything- we need a hypothesis.
In the world of tech products, we build in ways to measure what exactly how users interact with our products. We collect 100s of datapoints on every click. We then try to see if the users are able to figure out the best way to use our product and unlock full value.
But the answer to what we need to fix, does not come from just observing user clicks. We need to have a certain hypothesis of user behaviour and try to make the product intuitive for the user to understand. Also, if we don’t know our user well, no matter how much we measure and improve product design, we wont get it right. For example, if we are building a product for managing patient calls at a clinic, if we build it thinking it is the doctor who is going to use when it reality it is the receptionist who might use it, we would fail at creating a good user experience.
Similarly, if we don’t keep up with changes in user behaviour, we will again be ineffective. A good example of this the way Google Pay launched in India and made contactless payments a breeze. Every payment company out there had to re-look at their UX after that and match the user expectation of a 1 click payment. If one isn’t following these trends closely, the usage metrics will never go up.
So let’s use statistics to support and test our assumptions — but understand that they may not always be the ultimate truth. In order to make good decisions, we need numbers to support our assumptions but let’s not let them overwhelm us. The next time someone is quoting numbers, let’s step back and see the bigger picture once. This is the only way we can measure better and correct biases and stop the infinite loop of misinformation.
