2022 in the Rearview

By Adam Siegel on December 19, 2022

First off, thank you for taking the time to read our blog. Most of you probably got here from our newsletter, and I appreciate you being a subscriber.

I wanted to do a brief year-end retrospective on what we’ve been focused on both for your sake and for ours. Sometimes we take for granted how much we’ve accomplished in a 12-month window. Taking a moment to pause and look back helps us appreciate that better.

First, though, I wanted to share a few thoughts on where I think we are as a company within the area of forecasting.

Over the past 15 years of being in this space, I’ve been fortunate enough to watch crowdsourced forecasting go from a relatively unknown curiosity to an approach adopted by a wide range of organizations: government intelligence agencies, militaries, think tanks, investment banks, pharmaceutical companies, logistics companies, and many more.

When we first began working with these kinds of organizations, our value proposition was straightforward: we can create a better crystal ball than you have now. When the advanced research arm of the U.S. Intelligence Community, IARPA, funded a multi-million dollar research program “Aggregative Contingent Estimation” (ACE) in 2011, Dr. Phil Tetlock, Barb Mellers, and Don Moore further proved that you could indeed find “superforecasters” among a crowd and have an even better crystal ball. The legacy of that work, primarily Good Judgment’s cohort of superforecasters, continues to impress (and be quite accurate!) to this day.

While a better crystal ball will always be desirable, a parochial fixation on accuracy creates landmines for the success of any forecasting effort. Among them:

  • In politically charged environments, it’s hard to convince someone to take the risk of not being an accurate forecaster;
  • The credibility (and value) of the effort is always fragile: you’re good - until you make one bad call.
  • Organizations don’t always know what to do with the specificity of a probabilistic forecast as part of their decision or policy making processes. This diminishes the value of having highly accurate forecasts.
  • A focus on accuracy comes at the detriment of soliciting early signaling from forecasters that is telling regardless of final outcome, such as interesting/contradictory rationales, or forecast updating and collaboration as change happens.

With this in mind, we spent a lot of time this year re-thinking what our true value proposition is to our clients. At our core, we are a forecasting company, so accuracy is always going to be a metric by which we measure ourselves. But what has been increasingly apparent for several years now is that the process of making people forecast and the outputs along the way - at least when talking about internal efforts - have more value to optimizing decision making than only striving to achieve the most accurate forecasts.

We’ve always known, for example, that our clients have a keen interest in the rationales provided alongside our forecasts as context to answer why a crowd forecast is what it is. Asking people to update their forecasts over time has also historically provided helpful trend data, and breaking down long-term 10+ year forecast questions into shorter-term guide posts has brought more rigor to questions typically impossible to forecast.

So this year, in partnership with our clients, partners, and contractors, we began to lean in to extracting more from the kinds of insights we could glean from the forecasting journey itself. For example, we:

  • Built upon the work of Dr. Phil Tetlock and J. Peter Scoblic in their Foreign Affairs article and Michael Page’s work at Georgetown’s CSET during our Foretell project, to formalize and implement on several of our projects an approach for “issue decomposition,” which seeks to create a shared view of the scenarios, lynch pins, and signals representing the outcome of a “fuzzy” forecast question.
  • Built new visualizations, sorts, and filters that would expose both contrarian thinking and the most “well-written” rationales to both buttress and challenge status quo thinking.
  • Introduced several new types of forecasting questions to better collect data from forecasters, including continuous questions (e.g. Will X happen in the next 6 months) and multi-time period questions (e.g. What will sales be in each of the next 4 quarters?)
  • Launched a "pre-mortem" input to the forecast rationale to guide forecasters to challenge their own thinking (and give analysts valuable assumption-challenging fodder) to consider why they might be wrong.
  • Created a new area to support analysts on individual forecast questions with forecast distributions, forecast update trends, and other insights.
  • Produced a new heuristic for the crowd forecast’s “reliability” by benchmarking against all other forecast questions.
  • Generated weak signal identification tools, including forecast cluster visualizations.
  • Expanded API specification to accommodate more individual forecaster analysis.
  • Overhauled forecaster user profiles to reveal personal performance statistics and visualizations, recent activity, and earned badges.

Of course the lifeblood of all our projects is the forecasters themselves, and we continued to evolve how we engage them. Notably, no one size fits any forecaster population, so we continuously learn to vary our techniques and craft approaches that can be a two-way street to incentivize participation. We get an individual’s judgment on a forecast question, and they get benefit back, whether it be from direct monetary compensation, improving their critical thinking skills, or knowing they’re serving a broader purpose to improve decision making in the organization they work for or support.

If the lifeblood of our projects is our forecasters, the lifeblood of our company is our clients. We were incredibly fortunate to work with leading organizations from around the world who did some amazing things this year. They were government intelligence and security agencies trying to bring more rigor to their analytic processes, threat assessment agencies trying to prioritize resources by quantifying risks to their countries, diplomatic agencies and the think tanks that advise them forecasting the likely outcome of pivotal events, pharmaceutical companies trying to optimize their drug portfolio and bring life-saving drugs to market sooner, consumer product companies trying to assess the future health of their supply chains, logistics companies leveraging their sales and product teams to forecast future container shipping costs, and media organizations striving to create new engagement models with their consumers. One of our clients, the Bertelsmann Foundation and Bertelsmann Stiftung, even showcased their forecasting platform, RANGE, at the last G7 meeting in Munster, Germany.

We’re also fortunate to have partners and contractors who both inspire us and make what we do for our clients that much better. The team at Good Judgment Inc, whom we provide with the forecasting platform for the Good Judgment Open and work with on a select number of clients for internal forecasting, has been a collaborator for years. We worked with Pytho this year to define a forecaster curriculum for forecasters on our INFER project. Pytho also taught a class to our INFER “Pro Forecasters” on open source research techniques that was well received. We had interns from Northwestern who have been invaluable helping on a variety of projects, and Katie Cochran, who has been instrumental in helping us think through our approach to issue decomposition, dashboard designs, and is just a wonderful sounding board and teammate in general. We call her a contractor, but she essentially feels like part of the family.

Speaking of family, I’m grateful to say that our team continues to grow; we brought on talented people across multiple new positions this year. While I personally remained in the midwest, our center of gravity has left Chicago (sadly) and is now in Washington, DC. With such a dramatic shift and a relinquishing of our physical office during the pandemic, I was concerned how we’d work with one another. And yet I think we’d all agree (as our 360 review process proved out), we’re content with the chemistry and working relationships we continue to have. For being such a quantitatively driven company, one of our main hiring philosophies is a very qualitative one: “no assholes.” We insist we all treat each other like grownups. We have no “office politics” or “palace intrigue.” We are passionate about our mission and do the work to be done, then go home and be whomever we are away from the company.

This year also saw us continue to mature our business processes - a major goal of ours for the past 3 years as we slowly scale the size of our company and work on more mission critical forecasting efforts. Most notably, we were audited under the SOC 2 Type II standard, which was an internal controls report capturing how we safeguard customer data and how well those controls are operating.

Finally, I’d be negligent if I didn’t give thanks to my co-founder, Ben, for another great year. Our company is on excellent financial footing, and we yet again increased our year-over-year revenue and profits from 2021. When I did Y Combinator years ago, they always said your co-founder will become like a spouse, and this is absolutely true. Ben and I talk multiple times a day, we make every important decision together, and I feel more confident about any discussion if he is there. I trust him implicitly. He is not only a wonderful engineer and friend, but a humble, under the radar thought leader in the forecasting space that should get more recognition than he does. He is the backbone of so much of what Cultivate does.

Despite a successful year, I am already excited about 2023. The INFER project run by UMD’s ARLIS will be announcing a co-sponsored forecasting tournament with a significant player in the AI space next month, and will also be formally launching a U.S. Government-wide forecasting platform in the spring. Our 6-12 month product roadmap will see a re-launch of our native mobile apps, enhancements to team and forecasting functionality, and much more. Potential new clients will see us take our crowdsourced forecasting methodology into an increasing number of industries where the additional rigor we provide for their analytic processes can lead to competitive advantage through lowered risk and smarter resource allocation.

Thank you for reading this, and as always, feel free to reach out to me with any ideas, questions, or critiques. I’d love to hear from you.

Happy Holidays and a Happy New Year,

Adam

general