Base Rates for Forecasting and Decision-Making
By Sean Kucer on October 18, 2021
To see the future, one must first see the past. This is the idea behind the “base rate” (also commonly referred to as the reference class or outside view). Base rates are a hallmark of good forecasting and decision-making, and the most accurate forecasters make better use of them than their less accurate peers. (1) Base rates are simply the rate of occurrence of a particular outcome in similar situations. If in past years 1% of people were software engineers, then the base rate (of a random individual being a software engineer) is simply 1%.
While the above may seem obvious, people tend not to incorporate base rate information when contemplating the future, or making decisions. Taking our above example, let’s say you see someone sitting at a computer in a coffee shop with fancy (geeky) headphones and a Google t-shirt, typing away. You might automatically assume they are a software engineer. But the base rate of any given person being a software engineer is only 1%. If you assume that they are a software engineer, that implies you think there is at least a 51% chance that they are. But does wearing expensive headphones and a Google shirt make it 50 times more likely that someone codes for a living? Probably not.
In this situation, it doesn’t matter that you mistake a stranger for a software engineer, but in many high-stakes decisions, people still fail to incorporate base rate information—leading to real-world negative consequences. For example, a study conducted by Alexei Milkov at the Colorado School of Mines found that petroleum explorers consistently overestimated the quality of the fields they discovered by neglecting base rates. (2) The petroleum explorers were mistaken about how lucrative their fields were, and habitually under-delivered on output and value. In contrast, two different studies focusing on the Return on Investment (RoI) of offshore wind farms found that the use of base rates in planning the projects increased the probability the projects were delivered on time and under budget. (3) Noting similar results, The American Planning Association, a professional organization representing the field of urban planning (note: large, capital intensive projects) has even officially endorsed the use of base rates as part of any planning exercise. (4)
So offshore wind farm operators and urban planners can use base rates to improve their cost estimation, but how can we begin applying the use of base rates to the kind of probabilistic forecasting we do on the Cultivate platform? Say we are trying to determine if the Democrats will still have a majority in the House of Representatives after the next election. On average over the past 50 years, the Democrats have held about 240 seats in the House--this is one base rate. But the 2022 election is a midterm where the majority party has the White House, and we have the well-established base rate that the party in control of the White House generally loses seats in midterms (and sometimes a lot of seats). (5) You could narrow this even further, and look at the number of seats held by the incumbent party in midterms when the president has an approval rating lower than, say, 50%. The key here is identifying similar situations in the past and treating them as the baseline expectation for what will happen in the future.
And that is really all there is to base rates at the most basic level. To use base rates yourself, try following the checklist below:
How to use base rates, a checklist
- Identify what situations from the past are analogous to the situation you want to better understand or forecast on (e.g. total cost of a similar project).
- Identify the relevant statistics from these analogous situations (e.g. determine project costs and overruns, time to delivery, or even the ultimate return on investment for your pool of comparable projects).
- Consider which base rates are most informative and most analogous to the situation you face, and weigh these higher than other base rates (e.g. the cost of more recent projects may be more analogous than the cost of one from 10 years ago).
- Combine the base rates you have found into an overall probability. For example, if you had two base rates—one of 25%, and one of 10%—you would average these to 17.5%. But, following step 3, you may find that the first one is more analogous and informative, and so you could give it more weight (for example, by bumping the 17.5% up to 20%). This would be your starting probability, which you would update with further techniques that we will explain in later blogposts.
- In your final forecast and/or decision, apply the base rates you have found and combined. This is the most important step, as it is surprisingly easy to go through all the work of establishing good base rates and yet forget to apply them when it matters. If you’d like to get some practice on this right away, sign up to forecast on matters of national security on CSET’s Foretell.
Good use of base rates improves forecasting, but most importantly, improves decision-making. As we aim to stress throughout this series, good forecasts have no inherent value—-they have value only insofar as they can be used to improve important decisions. This is what we do at Cultivate: improve your forecasts and then help you apply them to the decisions you want to improve. But you don’t have to wait to set up a forecast platform to make better decisions—just start using base rates!
-----
Sources:
(1) https://www.edge.org/conversation/philip_tetlock-edge-master-class-2015-a-short-course-in-superforecasting-class-iv
(2) https://pubs.geoscienceworld.org/aapgbull/article-abstract/101/12/1905/523089/Integrate-instead-of-ignoring-Base-rate-neglect-as?redirectedFrom=PDF
(3) These studies, and others with similar findings, can be found on page 40 at https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/983759/updating-the-evidence-behind-the-optimism-bias-uplifts-for-transport-appraisals.pdf
(4) https://www.pmi.org/learning/library/nobel-project-management-reference-class-forecasting-8068
(5) https://fivethirtyeight.com/features/republicans-2020-gains-in-the-house-set-them-up-well-for-2022/
