It doesn’t take a great prognosticator to see that 2016 will go down in the history books as a bad year for their profession.
The UK voted to leave the EU. The Cubs won their first World Series title in 108 years. And almost every forecaster in the business was caught by surprise with Donald Trump’s victory in the presidential election.
“I think, right now, we’re sort of groping in the dark and I think the events of 2016 certainly underscored that,” Philip Tetlock, a professor at the University of Pennsylvania’s Wharton School, told an audience at the World Economic Forum in Davos, Switzerland, this past January.
Critics argue the elite status of Davos attendees renders them tone-deaf to the concerns of the common person. The sentiment’s been echoed for some time, especially given their mixed track record of predictions, highlighted in this recent Bloomberg report. However, the charge was particularly topical this year since most Davos attendees failed to predict the watershed political moment in 2016. Then again, who did? Even the best at predicting, known as “superforecasters,” didn’t foresee a Trump win.
These superforecasters don’t have a superhero-like origin tale. The term was born out of a prediction competition run by the Intelligence Advanced Research Projects Activity (IARPA) in 2011. In the tournament, five university teams made 500 predictions about a variety of topics over four years, ranging from disease outbreaks in sub-Saharan Africa to toppled dictators in the Middle East.
Tetlock’s team from Penn, called The Good Judgement Project, ranked in the top two percent of thousand that participated. In fact, their forecasters were 30 percent more accurate than those made by the intelligence community. Tetlock continued the experiment in forecasting with public and private spin-offs, Good Judgment Open and Good Judgement, Inc. And that’s how the superforecasters were born.
“People often think Superforecasting sounds cool, a blend of Sherlock Holmes and ‘M’, maybe dressed in a cape. In reality, I think most Superforecasting is done quietly in our living rooms, late at night intensely pondering the news instead of watching TV,” one superforecaster named Kate Patterson tells RealClearLife.
Tetlock argues in his book, Superforecasting: The Art and Science of Prediction, that anyone can achieve that level of accuracy if they have a decent grasp of probability, are aware of their pre-existing biases, and are determined to improve. Superforecasting is “a product of particular ways of thinking, of gathering information, of updating beliefs,” Tetlock writes in the book.
It doesn’t take a Ph.D. in political science, either. David McCullough was an underwater archeologist. Elaine Rich was a pharmacist. Marc Koehler used to work at the U.S. State Department. Scott Eastman is a photographer and lecturer who lives in Romania. Now, they’re all superforecasters.
Do the same rules of forecasting apply with Donald Trump as President? Superforecasters say yes. (Andrew Harrer/Bloomberg via Getty Images)
Patterson, who lives in Brisbane, Australia, had a background in business but joined the forecasting tournament as a fun way of engaging with the news. “Instead of just reading stories about things that have happened, I can participate to create forecasts that used to respond to risks. A good example is the Early Warning Project challenge that looks at the risk of mass atrocities around the world,” she tells RealClearLife.
But even superforecasters like Patterson were forced to contend with the accuracy of their predictions following the U.S. election. “I think people may feel a false sense of confidence because you can always find a news story online to support your opinion—even if your opinion is wrong,” Patterson says. “Then we are surprised when things don’t go that way.”
Despite the alarmist headlines of a volatile paradigm shift, forecasters don’t see 2016 as a watershed moment for the world. “I don’t think the world is getting more unpredictable,” Patterson adds. If anything, 2016 was a boon for superforecasters. Sure, the prediction may have been publicly tarnished, but it’s a reminder of the irreducible uncertainties that exist in the world.
Patterson argues that superforecaster’s predictions “are more useful for effective decision making because we show that there is inherent uncertainty in the world.” However unlikely an event may be, there’s still a degree that it may happen. “A superforecaster will very rarely say something is 100 percent—there are always unpredictable factors,” Patterson explained.
Fake news aside, the deep polarization in America means that many sources—both individuals and media outlets—are more skewed than they were before. As a result, forecasters have to be more picky about what news story they consider when making their predictions and how they did do it. “Many new sources and forecasts give a definite answer on what the future will be, ignoring other sides to the story,” Patterson says.
By choosing to absorb tangible matters, instead of chasing updates or “chiron consumption” as one superforecaster put it, information becomes much more substantive. Take the administration’s policy stance on the Arab-Israeli conflict, for instance.
Perhaps instead of making a conclusion from President Trump’s press conference with Prime Minister Netanyahu, take a step back and evaluate the situation over a few weeks or months. By asking a series of questions—Will the U.S. move its embassy to Jerusalem? Have they backed Israeli settlement expansions?—that can be answered with actions and not comments, drilling down to the core conclusion becomes a little easier.
“Oxford taught me to rely on data [over] gut instinct, look at multiple perspectives, and always consider the broader systems at play,” Patterson says. “These are the same principles I use for breaking down Superforecasting questions.”
While not every event on the planet is determined by American politics, the United States still is a major driver of geopolitical events. Still, a more spontaneous leader of the free world doesn’t necessarily mean the world is less predictable. Other nations can be expected to mitigate some risks they wouldn’t have done before the current political environment.
In the end, the events of 2016 caused many to questioned the value of predictions since the outcome is rarely controlled—and that’s a fair point.
“All forms of policy planning assume forecasting,” Tetlock told the same audience in Davos. “Anyone that has a policy preference on anything is making an implicit forecast, an implicit conditional forecast.”
The only real option, given the reality of geopolitics, is to improve forecasting as best as possible. That starts with making the act of prediction more transparent. It also means holding forecasters accountable for inaccurate predictions. The goal is to inform the general public, so they can find sorting good predictions from the bad ones a little more, well, predictable.
—Matthew Reitman for RealClearLife
This article was featured in the InsideHook newsletter. Sign up now.