Tracking the Outcome of Strategic Questions with Crowd Forecasting

By Adam Siegel on October 07, 2021

"Making better decisions starts with understanding this: uncertainty can work a lot of mischief." -Annie Duke

At Cultivate, we provide a technology platform to pose forecasting questions to large groups of people to try and quantify uncertainty and risk. These forecasting questions may be binary in nature, e.g. “Will the 2022 Olympics be cancelled?” or have multiple possible outcomes: “Which candidate will win the election?” At any given time, a participant can assess the probability of an outcome occurring, and we constantly factor their forecast into a consensus. Because the approach typically solicits input from large numbers of people, it’s often referred to as “crowdsourced forecasting.”

In business and government, crowdsourced forecasting has proven to address problems experienced in assessing uncertainty, where traditional prediction methods are impossible to deploy effectively. Sometimes historical data is lacking or unstructured, making predictive models difficult to construct. Experts who are hired to offer future-looking analyses introduce bias because of particular worldviews or mis-aligned incentives. And diversity of input is usually difficult to obtain because of inter- or intra-organizational politics, restrictions, and logistics.

One unfortunate shortcoming of how crowdsourced forecasting has been implemented thus far, however, is the tendency to focus on narrow outcomes rather than broader areas of uncertainty.

For example, organizations often get caught up creating the specific, falsifiable questions required for forecasting such as “Will a water emergency be declared in X geographic region before the end of 2022?” or “Will the U.S. announce new trade tariffs against the EU in response to the Carbon Border Adjustment Mechanism?” and fail to fully analyze and understand the broad, strategic question about what impact climate change will have on their operations.

The assessment of these kinds of detail-level forecast questions can be valuable input for analysts and managers, whereas senior decision-makers are defining strategy or policy, not implementing or assessing it. They require a more comprehensive view of the bigger picture issue.

There are qualitative methods designed to assess where future reality may be headed at a much higher level, such as scenario planning. First popularized by Royal Dutch Shell in the 1970’s, it is still in practice in various forms to this day.

Scenario planning is essentially an ongoing, thorough review of “what-if” situations, with potential courses of action created for each. Unlike a 3 or 5 year strategic plan which is a single, fixed bet on what the future will hold, scenario planning is broadly open-ended. It considers any number of possibilities and allows an organization to proactively plan for a range of outcomes at once.

Critiques of this approach often highlight the ease by which human bias and lack of continuous attention harms the process. Mckinsey & Company, in a 2015 article entitled “Overcoming Obstacles to Effective Scenario Planning” mentioned excessive optimism about certain scenarios, an over emphasis on unlikely events, and over relying on historical precedent as key mistakes organizations make implementing a scenario planning process. Executives may also use scenario planning as a one-time “boot camp” style exercise to inform a strategic plan, but then never revisit it. The scenarios they worked so hard to originally define quickly lose relevance as ground truth outpaces them.

A New Approach

We’re often asked by our clients how to use crowdsourced forecasting to predict long term outcomes. Rather than simply asking a forecast question about that outcome, our advice has always been to look for the “building blocks” of what will make that future reality come to fruition. What are the signals that one could look for that would indicate a lower or higher likelihood of the long term outcome taking place?

This same type of thinking could be applied to a strategic question an organization has, like “How prevalent will alternative payment methods such as cryptocurrency become in the next 3 years?” To forecast such a “fuzzy” question, we can “decompose” it and look for the pivotal factors that would lead us to better understand the likelihood of one outcome over another. In essence, we can combine some processes of traditional scenario planning with the quantified results of probabilistic crowdsourced forecasting.

Advocates of both approaches are already leading the charge in this direction. In a recent article entitled "A Better Crystal Ball" in Foreign Affairs, Peter Scoblic and Phil Tetlock wrote:

There is a better way...It involves reconciling two approaches often seen to be at philosophical loggerheads: scenario planning and probabilistic forecasting. Each approach has a fundamentally different assumption about the future. Scenario planners maintain that there are so many possible futures that one can imagine them only in terms of plausibility, not probability. By contrast, forecasters believe it is possible to calculate the odds of possible outcomes, thereby transforming amorphous uncertainty into quantifiable risk. Because each method has its strengths, the optimal approach is to combine them. This holistic method would provide policymakers with both a range of conceivable futures and regular updates as to which one is likely to emerge. For once, they could make shrewd bets about tomorrow, today.

A Georgetown think tank and Cultivate partner, the Center for Strategic and Emerging Technologies (CSET), has planted the seeds of this kind of approach on their public crowd forecasting site Foretell, where they’re exploring the future contours of the relationship between the U.S. Department of Defense and Silicon Valley. In preparation for issuing analysis of the issue, CSET is working with Subject Matter Experts to define the “sign posts” that would indicate directionality for that relationship (“Will it expand? Will it contract? Will it become more hostile? Friendlier?”), then crafting those sign posts as forecast questions, for which a diverse, global community of interested participants are submitting probabilistic assessments. In this way, CSET is realizing what Scoblic and Tetlock proposed: providing policy makers with plausible future-world outcomes, and near real-time quantitative assessments of how likely each of those outcomes are.

Inspired by these approaches, in a handful of recent projects in both the public and private sectors, we’ve been applying a similar approach we're calling, “Strategic Question Decomposition.”

Here’s how it works:

As noted in the steps above, once forecasting begins, results can be aggregated visually and/or mathematically to represent trends in sets of pivotal factors that may drive our understanding of the bigger picture in one direction or another. For example, here’s a strategic question decomposition for a large energy company trying to understand how green trends will impact their business:

Starting almost immediately, aggregated results of forecasts of the detailed signals (e.g., step 6 from above) can regularly be shared with senior decision makers or analysts in an interactive visualization, allowing them to explore the overall direction of each pivotal factor based on the individual signals - ultimately helping them understand where the strategic question they originally asked is heading.

Conclusion

The approach we’ve outlined in this blog post bridges the gap between the specific, falsifiable questions that are essential in probabilistic crowdsourced forecasting and the strategically-relevant questions that are of interest to decision makers. We envision a future capability to combine data-driven, algorithmic forecasts (e.g. forecasts based on sensor data) with human crowdsourced forecasts to give an even more comprehensive view of the direction pivotal factors are heading.

But no matter how innovative we can be to elicit and aggregate forecasts, and no matter how accurate we can become, our approach still hinges on one, very human characteristic: creativity. As powerful as our forecasting capabilities may be, we must still use our creativity and reasoning skills to identify all the pivotal factors and signals that present the possible paths of what future reality may be. Only then can we be confident in the results as input to our most important strategic decisions.

cultivate forecasts