How can we judge the accuracy of Nate Silver's predictions?

Question

Firstly, he gives probability of outcomes. So, for example, his predictions for the U.S. election is currently 82% Clinton vs 18% Trump.

Now, even if Trump wins, how do I know that it wasn't just the 18% of the time that he should've won?

The other problem is that his probabilities change over time. So on July 31st, it was almost a 50-50 between Trump and Clinton.

My question is, given that he has a different probability every day for the same future event with the same outcome, how can I measure how accurate he was for each day he made a prediction based on the information that was available up to that day?

Answer 1

Probabilistic forecasts (or, as they are also known, density forecasts) can be evaluated using scoring-rules, i.e., functions that map a density forecast and an observed outcome to a so-called score, which is minimized in expectation if the density forecast indeed is the true density to be forecasted. Proper scoring rules are scoring rules that are minimized in expectation only by the true future density.

There are quite a number of such proper scoring rules available, starting with Brier (1950, Monthly Weather Review) in the context of probabilistic weather forecasting. Czado et al. (2009, Biometrics) give a more recent overview for the discrete case. Gneiting & Katzfuss (2014, Annual Review of Statistics and its Application) give an overview of probabilistic forecasting in general - Gneiting in particular has been very active in advancing the cause of proper scoring rules.

However, scoring rules are somewhat hard to interpret, and they really only help in comparing multiple probabilistic forecasts - the one with the lower score is better. Up to sampling variation, that is, so it's always better to have a lot of forecasts to evaluate, whose scores we would average.

How to include the "updating" of Silver's or others' forecasts is a good question. We can use scoring rules to compare "snapshots" of different forecasts at a single point in time, or we could even look at Silver's probabilistic forecasts over time and calculate scores at each time point. One would hope that the score gets lower and lower (i.e., the density forecasts get better and better) the closer the actual outcome is.

Answer 2

In Nate Silver's book The Signal and the Noise he writes the following, which may provide some insight for your question:

One of the most important tests of a forecast - I would argue that it is the single most important one - is called calibration. Out of all the times you said there was a 40% chance of rain, how often did rain actually occur? If, over the long run, it really did rain about 40% of the time, that means your forecasts were well calibrated. If it wound up raining just 20 percent of the time instead, or 60 percent of the time, they weren't.

So this raises a few points. First of all, as you rightly point out, you really can't make any inference about the quality of a single forecast by the result of event which you are forecasting. The best you can do is to see how your model performs over the course of many predictions.

Another thing that is important to think about is that the predictions that Nate Silver provides are not an event itself, but the probability distribution of the event. So in the case of presidential race, he is estimating the probability distribution of Clinton, Trump, or Johnson winning the race. So in this case he is estimating a multinomial distribution.

But he is actually predicting the race at a far more granular level. His predictions estimate the probability distributions of the percentage of votes each candidate will garner in each state. So if we consider 3 candidates, this might be characterized by a random vector of length 51 * 3 and taking values in the interval [0, 1], subject to the constraint that the proportions sum to 1 for the proportions within a state. The number 51 is because other are 50 states + D.C. (and in fact I think it's actually a few more because some states can split their electoral college votes), and the number 3 is due to the number of candidates.

Now you don't have very much data to evaluate his predictions with - he's only provided predictions for the last 3 elections that I'm aware of (was there more?). So I don't think that there is any way to fairly evaluate his model, unless you actually had the model in hand and could evaluate it using simulated data. But there are still some interesting things that you could look at. For example, I think it would be interesting to look at how accurately he predicted the state-by-state voting proportions at a particular time point, e.g. a week out from the election. If you repeat this for multiple time points, e.g. a week out, a month out, 6 months out, and a year out, then you could provide some pretty interesting exposition for his predictions. One important caveat: the results are highly correlated across states within an election so you can't really say that you have 51 states * 3 elections independent prediction instances (i.e. if the model underestimates candidates performance in one state, it will tend to underestimate in other states also). But maybe I would think of it like this anyway just so that you have enough data to do anything meaningful with.

Answer 3

For any single prediction you can't, any more than we can tell if the claim "this coin has a 60% chance of coming up heads" is close to correct from a single toss.

However, you can assess his methodology across many predictions -- for a given election he makes lots of predictions, not just of the presidential race overall but many predictions relating to the vote for the president and of many other races (house, senate, gubnertorial and so on), and he also uses broadly similar methodologies over time.

There are many ways to do this assessment (some fairly sophisticated), but we can look at some relatively simple ways to get some sense of it. For example you could split the predictions of the probability of a win up into bands of e.g. (50-55%, 55-65% and so on) and then see what proportion of the predictions in that band came up; the proportion of 50-55% predictions that worked should be somewhere between 50-55% depending on where the average was (plus a margin for random variation*).

So by that approach (or various other approaches) you can see whether the distribution of outcomes were consistent with predictions across an election, or across several elections (if I remember right, I think his predictions have been more often right than they should have been, which suggests his standard errors have on average been slightly overestimated).

* we have to be careful about how to assess that, though because the predictions are not independent.