Humans do it all the time. Sometimes with great success. Not just easy-to-measure and profitable endeavors like sports betting or the stock market, but essential human interactions like, “what’s the best way to welcome a kindergarten student on the first day of school,” or “If we arrange the intersection this way, traffic will flow better.” In matters of public health and engineering, the ability to have a good idea about the repercussions of our work is urgent.
When dealing with a prognosticator, it’s worth asking three questions:
“What’s your track record?” It’s unlikely we’ll be right every single time, but once we adjust for luck and statistical anomalies, do you regularly outperform the others, or are you simply loud about it?
“Can you show your work?” It’s hard to trust someone who has a secret method. While this might be a competitive requirement, it’s more likely that the person has simply had a lucky streak (streaks are statistically likely).
“Have you taught your method to others?” This is a variation of the previous question. If people are using the method to successfully predict the future in other areas, then we’re seeing a resilient and robust approach to understanding how the world works.
Rules of thumb (the topic of my very first book, co-authored 34 years ago) are a stand in for the sort of rigor that is far more common today. With our predictions etched into the memory of the internet and more data available than ever before, we ought to be better at predicting what’s going to happen next and determining who’s good at that and who isn’t. But belief is a strong force, widely held, and sometimes it takes us a while to realize that confidence and volume are not a replacement for seeing things as they are and understanding how they work.