Superforecasting -- Everything has a base rate
By Cultivate Labs on October 01, 2015
In a recently released book titled Superforecasting, Phil Tetlock of the Good Judgement Project (a client & partner of Cultivate Labs) goes into depth about what makes a person a "Superforecaster". The book goes into many aspects of forecasting, such as what makes a good question, how to score forecasts, techniques these Superforecasters use to create their accurate forecasts, and why being a pundit on tv is mostly about being vague with what you say so you always have an excuse if your prediction turns out wrong.
As much fun as it is to talk about the blowhards on cable tv, I want to focus on one of those techniques talked about in the book that Superforecasters use that can quickly turn your wild guesses into decently-informed guesses.
Tetlock calls it the inside / outside view technique. He argues that no matter what event they’re trying to forecast no event is entirely unique and that finding similar events to the one in question is a great way to make forecasts. Basically, we want to find the base rate of similar occurring events, and then adjust that percentage by taking into account the differences between this event and the similar ones.
A great example of this is the current 2016 US Presidential election, and how our prediction market has Donald Trump at 3% to win the general election even though he’s doing well in pretty much every poll.
To see if this is a reasonable probability for Trump, we can look for the base rate of candidates I’ll choose to call, “Candidates that are good with the media and give them something to write about but, let’s be real, could never be president”. And Trump is a grade-A CTAGWTMAGTSTWABLBRCNBP. Just like all the other candidates like Trump, you probably don’t remember them because they never become their party’s chosen candidate. In fact, they pretty much have a 0% success rate in this day and age. From a starting point like this, you’d go ahead and adjust your probability accordingly, giving Trump a slim chance of breaking the mold and winning.
Another relevant example of using the inside / outside technique involves the NFL, and predicting the probability of a certain team making the playoffs after losing the first three games of the season. This season there are 4 teams that have managed to be perfectly defeated through these three games -- Detroit, Chicago, New Orleans, and Baltimore.
The outside view / base case here is how often 0-3 teams make the playoffs. In the history of the NFL under the current playoff rules, 2% of 0-3 teams end up playing more than 16 games. From there, we take the inside view and adjust that 2% based on the individual teams.
For example the Baltimore Ravens haven’t looked terribly bad and have been close in each of their three games. They’re also in a weaker division with one of the teams (Steelers) missing their starting quarterback for a couple months. Combined with a schedule that seems pretty weak for the rest of the season, it’s not impossible for them to make the playoffs. Keep in mind that we’re trying to put a number on the event happening and not use fluffy words like “not impossible”. In this case, you might think of bumping the probability of the Ravens making the playoffs to something like 8%-10%. Much higher than the 2% base rate, but still improbable.
On the other hand, I’d bump the Bears chances of making the playoffs down because 1) they're using their backup quarterback, and 2) they can't score points (punted all 10 possessions yesterday against the Seahawks) and were shutout for the first time since 2002. I'm a big Packer fan and would totally love to keep going about the terrible year the Bears are in for this year, but considering I don't want to make enemies of any more Bears fans, I'll refrain from listing more reasons. Safe to say, I don't think the Bears will make the playoffs this year.
The book outlines other techniques and methods that Superforecasters use, so don’t think this is the only useful piece of information to be gleaned from the book. This post isn't titled “One Method That Will Have You Predicting the Future in No Time!” Clickbait is annoying and in fact, one of the main messages of the book is that becoming a good forecaster takes a lot of practice, and that simply reading a blog post won't make you super. So check the book out, and use that knowledge to start making better forecasts.
