New whitepaper: A roadmap for using AI in crowdsourced forecasting

By Vanessa Pineda on November 06, 2023

We previously wrote about our experiments with ChatGPT to help generate summaries of forecast rationales for the reports we deliver to analysts and decision-makers. A once time-intensive and manual process of sorting through hundreds of rationales became hyper-efficient with AI. Most recently, we integrated AI-generated rationale summaries powered by Claude into Cultivate’s forecasting platform. We’ve seen promising results as we continue to explore how we can leverage generative AI, specifically large language models (LLMs) such as ChatGPT and Claude, to strengthen the different aspects of crowdsourced forecasting.

Our latest whitepaper, A Roadmap for Using AI to Create Value in the Crowdsourced Forecasting Lifecycle, outlines recommendations based on the past few months of testing and implementing AI within four areas that are essential to running a crowdsourced forecasting program: question development, reporting, engagement, and forecasting. 

(It’s important to note that we ran our AI experiments using anonymized data from our public platforms. The recommendations in the paper may not be feasible to implement for an organization running an internal forecasting program due to privacy concerns with AI, yet many organizations are creating their own AI chatbots which would make much of this possible.)

Below are some of the main takeaways. 

1. AI can be used to run through an ‘issue decomposition’, but AI is not yet capable of authoring proper forecast questions. 

Our tests showed us that AI could, at a basic level, be coached to go through an issue decomposition, Cultivate’s methodical approach to decomposing complex strategic questions into key drivers and forecastable signals. We found that AI could help us do initial strawmans of a decomposition when given a strategic question to break down. Although AI can't provide a complete picture of the issue, it certainly saved some research time and generated a draft to react to and expand upon with our own team and experts. Similarly, a nearly complete decomposition drawn up with experts could be fed to AI to detect if experts missed any important drivers or signals to consider. However, when it came to the more precise matter of authoring a falsifiable forecast question suitable for crowdsourced forecasting, AI didn’t come close as it was prone to factual errors and vague resolution criteria for questions. 

2. AI can reliably generate summaries of crowd rationales and detect patterns from data trends.

Given the language processing skills of LLMs, it was a natural first step that we make AI part of our reporting process by using it to generate thematic summaries of the crowd's qualitative data. We've had success using AI  to reliably parse out and produce from the data the main arguments for and against a certain event’s outcome. To expand on our use of AI in reporting, we explored and saw early promise in using it to pull other analytics from rationales to help contextualize forecast trends, such as detecting trending terms, shifts in the crowd forecast based on changing events or news cited by forecasters, and early warnings and risks indicative of a certain “bad” outcome. We were particularly impressed with AI's ability to pick out a list of major points of disagreement among the crowd around a specific question, as well as its ability to pick out coherent and relevant reasons around key shifts in the crowd forecast. For example, on a day where the crowd forecast decreased by 10% on question, AI could summarize specific events cited by users for lowering their forecasts.

3. AI provides solid ideas to build an engagement strategy, and can easily be used to assist with personalized communication.

User engagement is the lifeblood of sustaining a crowd forecasting program. Our tests in this area looked at how we could use AI at a strategic level to build an engagement strategy, and more tactically to develop personalized communications. We were able to coach AI on Cultivate’s approach to developing an engagement strategy, and it produced pretty "standard" best practice interventions for each of our engagement drivers (lead, develop, recognize, and communicate). Some interventions were reasonable to implement, and there were a few creative ideas that would be harder to produce in real life. For example, AI suggested emails to users promoting specific forecasting topic areas, running themed forecasting challenges, and setting up a  forecasting mentoring program. Once we saw that AI could help us build an engagement plan, we wanted to see how well it could implement one of the interventions: a personalized communication giving feedback to a user about their forecasting performance. We found that AI was effective (with oversight from an experienced human editor) to assist with producing personalized communications tailored to such specific goals and audiences.

4. AI can facilitate higher quality outputs from forecasters and reinforce forecasting best-practices. 

A forecasting program's value is directly related to the quality of the forecasts that the crowd is able to provide. We experimented with how AI could be integrated into effective forecaster education and development in order to improve overall forecast quality. Our goal was to determine ways AI could assist forecasters to help stimulate best-practice forecasting behavior, including question research (e.g., base rates and historical trends), thoughtful rationale writing, and updating forecasts. Not surprisingly, we found AI to be an excellent research tool for getting up to speed on a forecasting topic and the various perspectives on a possible outcome. AI also acted as a strong editor for rationales; when probed to find holes in a forecaster’s judgments surrounding a forecast, AI suggested perspectives to consider or pointed out areas of weakness in reasoning. Another way to use AI in forecasting we found beneficial was to use it to evaluate stale forecasts and help make updates by assessing the crowd's recent assessments. 

If you’d like to read more about our AI experiments, limitations we stumbled into, and detailed recommendations for use cases, visit our Insights page to download the new whitepaper.

Special thanks to Valerie Lippin for contributing to this post.