New Type of Forecasting Interface Creates a Lot of Options
By Zach Siewert on August 31, 2021
We’re excited to announce the launch of a beta version of a new forecasting capability on the Cultivate platform we’re calling “multi-time period questions.” This new type of forecasting input now gives us the ability to:
- Collect new data in the form of numeric forecasts (e.g. “I think we will sell 50,000 widgets this year”);
- Ask the forecaster to provide inputs across 2+ time periods (e.g. “What will revenue be for Company X over the next 4 quarters?”
- Pre-load data to show the forecaster what the historical trend has been to assist in their first forecast;
- Enter forecasts by drawing directly on a graph;
- Collect 90% prediction intervals (i.e. “I am 90% certain the outcome will be between x and z, with my exact forecast being y”)
Providing Exact Numeric Estimates vs. Ranges
Up until now, a forecaster has always been asked to provide a probability estimate for any question we ask. This is fine for most questions, but is less than ideal for others. Consider a question like: “What will Amazon’s revenue (in USD billions) be next quarter?”
Typically, we would structure this question like the example below:
This achieves the desired purpose, but can leave forecasters wasting time fussing over each numeric range, trying to fit their own estimate to the predefined answers. It also doesn’t provide a level of specificity some may want to collect from their forecasters.
Now, our team has taken the burden of assigning probabilities to predefined numeric ranges and moved it behind the scenes. We now allow forecasters to enter a point estimate for their forecast along with a 90% prediction interval (a type of confidence interval), all in an easy-to-use chart format:
A best practice in forecasting is to consider the base-rate, or historical pattern and precedent for what is to come in the future. Instead of hiding this kind of information in the background, we can now display it to forecasters in an integrated view. For question authors, this is a simple .csv upload with the data.
Forecasting Multiple Time Periods
Better yet, as you can see below, by moving input to a graphical format, we can collect forecasts across multiple time periods. Suddenly instead of having to ask 2+ questions in tandem or sequentially (“What will revenue be in Q1?” “What will revenue be in Q2,” etc.,) we can collect those forecasts in a single shot:
Optionally, as time elapses, the question can be configured to automatically add the next time interval, so the period of time being requested can continuously “roll.” Referring to the same example above, once Q3 2021 passes on the calendar, Q3 2022 would automatically be added and forecasters can update their graphs with their new estimates for the time period.
So how does scoring work?
When creating a new question, the author creates their own numeric ranges for scoring purposes, then our algorithm takes the forecaster’s prediction intervals and maps them to the ranges. Importantly, the forecaster’s prediction interval needn’t be symmetric, but can skew to better reflect their view on the question. With this mapping, the question can be scored as usual, providing a Brier (and relative Brier) score when the question resolves. For a refresher about our scoring system, check out this article from our Crowdsourced Forecasting Guide.
Multi-Time Period Questions in Action
This new question type opens up an array of new forecasting use cases we’re excited for our clients to try:
- Run an equity analysis competition with simple entry and automatic, objective scoring, e.g. "What will Amazon’s revenues be over the next 4 quarters?" or "How many Cybertrucks will Tesla ship in each of the next 4 quarters?"
- Forecast the course of Covid, e.g. "What will the total number of hospitalized Covid patients be in each of the next 6 months?"
- Project multiple releases of a statistic within one question, whether it’s GDP growth, sales, or even poll results, e.g. "What will the YoY CPI Inflation rate be each of the next 6 months?" or "How will the percentage of U.S. residents who do not trust the military change over the next 3 years?"
Try it out
A beta version of this new type of forecast input is live now at Foretell, CSET / Georgetown’s site to forecast security and technology issues.
You can read more about how they’re using them here, and try making a forecast using the new interface on this question.
Please let us know if you have an opportunity to make a forecast and how you felt about the process!
