How crowdsourced forecasting works

By Cultivate Labs on March 04, 2024

What is crowdsourced forecasting?

Crowdsourced forecasting is the process of soliciting quantitative forecasts (e.g. probabilities) about future events from a large group of people, then aggregating them into a "crowd" forecast. By harnessing the collective intelligence of a large, diverse group of people, crowd forecasting gives an analyst or decision maker valuable outside perspectives to consider.

Despite the “forecasting” in the name, our primary focus is not to achieve a better “crystal ball” (although, the end result is often improved accuracy). Instead, crowd forecasting is designed to strengthen and reinforce existing decision analysis processes, and create a “challenge environment” that encourages continuously reassessing assumptions and conclusions prompted by the feedback loop from ongoing crowd forecasts.

What is the history of crowdsourced forecasting?

  • 1907: The “wisdom of crowds” phenomena first discovered by British statistician Sir Francis Galton who observed the median guess of the weight of an ox – collected from 800 people at a fair – was within 1% of the right answer.
  • 1988: The University of Iowa launched the Iowa Electronic Market to let people bet on the outcome of elections.
  • 2004: James Surowiecki publishes “The Wisdom of Crowds,” which popularized the concept in business settings.
  • 2011-2015: The IARPA-ACE project was a baseline condition for seminal research by the Intelligence Community (IC) showing crowdsourced forecasting methods -- using only unclassified, publicly available data can increase forecast accuracy by as much as 30% over intelligence analysts with access to classified information (Washington Post)
  • 2014: PredictIt, a New Zealand-based online prediction market offering exchanges on political and financial events, was launched. By 2016, the website had approximately 29,000 active traders.
  • 2015: Good Judgment Open launches, creating a public forecasting tournament designed with the same forecasting processes the winning Good Judgment team leveraged in IARPA-ACE.
  • 2016: During the 2016 U.S. elections, PredictIt, along with other prediction market websites, received attention from various media outlets (e.g. PBS).
  • 2020: The launch of INFER (formerly Foretell), a public crowd forecasting site sponsored by RAND and formerly sponsored by Georgetown’s Center for Security and Emerging Technology (CSET) and the University of Maryland’s Applied Research Laboratory for Intelligence and Security (ARLIS). INFER decomposes technology and geopolitical security related policy issues into forecastable propositions and elicits forecasts from a global pool of forecasters.
  • 2020-Current: Working with the UK Cabinet Office, Cultivate Labs implemented Cosmic Bazaar, a low-side crowd forecasting site for security-related agencies across the UK Government.
  • Current: Several public forecasting sites exist, such as Good Judgement Open, Metaculus, RANGE, and INFER. Many internal crowd forecasting programs exist within organizations, including Google, Broadspectrum, Shell, Abbvie, Procter & Gamble, and more.

In recent years, there has been a shift from seeing crowd forecasting as a highly accurate Magic 8 ball – to a process that reinforces and bolsters existing analytic processes and creates a feedback loop from ongoing crowd forecasts. Learn more about the value of crowd forecasting for government and industry. 

What are the outputs for analysts and decision makers of this process?

Outputs vary based on the forecasting focus and goals of the organization, but have historically included:

  • Forecast data/trends
  • Summarized rationales and commentary
  • Accuracy scores
  • Measure of bias and calibration
  • Executive dashboards and reports to support ongoing monitoring of the issue’s directional movement and related decision-making (see an example of an INFER January snapshot report for U.S. government stakeholders highlighting Iran’s progress towards a nuclear weapon with a heat map indicating momentum)
  • Curated workshops to define the client’s strategic issue, identify the influencing drivers and signals, and ultimately, craft relevant forecast questions (learn more about our issue decomposition workshops)

How do I harness this concept for my organization?

Cultivate can help your organization set up a crowdsourced forecasting program to inform your key strategic and operational decisions. Here’s are the typical steps:

  1. It starts with you. Tell us: What version of the future do you need to structure your policies and decisions?
  2. We will work with you to determine the drivers and signals/signposts that reflect that version of the future.
  3. Ongoing crowd inputs, including forecasts, rationales, and commentary are solicited and provided from a crowd of your internal teams or via one of our partners who runs a public forecasting platform.
  4. Forecasts and rationales are converted into outputs, including forecast data/trends, summarized rationales and commentary, accuracy scores, and measurements of bias and calibration, among others.
  5. Outputs are composed into live dashboards and executive reports to support analysis and decision-making.
  6. Equipped with forecasting data and outputs, you can continue to monitor your crowd’s forecasting activity for new updates to your critical strategic issues, or return to STEP 1 to adapt your version of the future and garner new or additional insights from your crowd.

Here’s a typical forecasting lifecycle:

For More…

We invite you to read more about Cultivate Labs and our methodology, including our issue decomposition process, and our enterprise technology, Cultivate Forecasts.

Check out some recent case studies about how our methodology is being used by numerous governments and businesses for a variety of use cases:

If you’re interested in learning more about how crowdsourced forecasting can improve your strategic decision-making, get in touch with us at [email protected]. 


Cultivate Labs crowdsourced forecasting fundamentals