A Brier Score is a way to judge and score the accuracy of probabilistic forecasts. For instance, if I say there is a 90% chance that the Cubs will win the World Series in 2017 (a probabilistic forecast), and then they subsequently win the World Series, I was "mostly" correct. But I also implicitly said that there was a 10% chance that the Cubs would not win, so that 10% allocation was "wrong." How do I quantify how "correct" my 90%/10% forecast was? 

A Brier Score gives us a way to score these forecasts so that we can compare their accuracy.

Calculating a Brier Score

You'll often see a Brier Score referred to as the mean squared error of a forecast. Let's break that term down and by showing how a Brier Score is calculated.

Let's say I made my 90% Cubs win forecast 1 week before the World Series ended (with the Cubs winning). We start by calculating the "squared error" for each of the 7 days that my forecast was valid. To calculate the squared error, we simply square the difference between my forecast and the actual outcome (100% if the Cubs win, 0% if they lose):

(0.9 - 1.0)^2 = 0.01

And for my implicit 10% Cubs lose forecast:

(0.1 - 0.0)^2 = 0.01

So my total error for the day:

0.01 + 0.01 = 0.02

To calculate the mean squared error, calculate the squared error for each day that my forecast was running and average them:

Day	Day 1	Day 2	Day 3	Day 4	Day 5	Day 6	Day 7
Forecast (win, lose)	0.9, 0.1	0.9, 0.1	0.9, 0.1	0.9, 0.1	0.9, 0.1	0.9, 0.1	0.9, 0.1
Error	0.02	0.02	0.02	0.02	0.02	0.02	0.02

We then take the average (aka mean) daily squared error to get a final score. In this case, since all of the days are the same, my mean squared error ends up being equal to 0.02.

Updating a Forecast

What would have happened to my Brier Score if I had updated my forecast mid-way through the week? Maybe the Cubs won a game on Day 3, leading me to update my forecast to 95% on Day 4. For any days where I had a 95% forecast, the squared error would be:

(0.95 - 1.0)^2 = 0.0025

And my new, implicit Cubs-lose forecast of 5%:

(0.05 - 0.0)^2 = 0.0025

So my total error for those days:

0.0025  + 0.0025 = 0.005

So my daily squared error would look something like:

Day	Day 1	Day 2	Day 3	Day 4	Day 5	Day 6	Day 7
Forecast	0.9, 0.1	0.9, 0.1	0.9, 0.1	0.95, 0.05	0.95, 0.05	0.95, 0.05	0.95, 0.05
Squared Error	0.02	0.02	0.02	0.005	0.005	0.005	0.005

So my mean squared error would be:

(0.02 + 0.02 + 0.02 + 0.005 + 0.005 + 0.005 + 0.005) / 7 = 0.0114

In the world of Brier Scores, a lower score (ie. less error) is better, so by updating our forecast mid-way through the week, we improved our score from 0.02 to 0.0114.

Next up: What are Relative Brier Scores and how are they calculated?

---

Go back to The Ultimate Guide to Crowdsourced Forecasting and Prediction Markets.