Introducing un-scored questions
By Adam Siegel on October 15, 2024
The religion of probabilistic forecasting says SCORE EVERYTHING and we try to(!), but there are times when you want to ask a question that may not have good resolution criteria (e.g. asking retroactively who committed an attack if you already know the definitive answer will never be provable), or the answer is 5+ years from being known.
In these instances, scoring the question for accuracy either isn't possible, or will take a very long time. Yet analysts or decision makers consuming this information still want people's quantitative, probabilistic input.
For these situations, we've recently introduced un-scored questions.
To a forecaster, the experience is exactly the same. You enter your probability, your rationale, and you're done. As always, you can update your forecast too whenever you want. When the question timeline has expired, it will simply end and it will never be scored.
Cosmic Bazaar, the UK Government's crowdsourced forecasting program, has been using these to great effect to capture input from forecasters about a range of longterm questions in preparation for preparing analytic products for policymakers. They ask questions whose results won't be known for 10, 20, 30 years in the future and only allow input for 1-3 months, at which point the question is closed.
The benefit of course is that you're leveraging your crowd in a different way. These are people, in Cosmic Bazaar's case, who may have been forecasting now for 4+ years and have a track record. Logic would dictate that if they've done well, it's also useful to listen to them in other contexts.
For administrators: when you're publishing a question, you can simply uncheck a new checkbox that says "Calculate Accuracy Scores." Everything else would be set as you would for a scored question.
