In Nate Silver's book The Signal and the Noise he writes the following, which may provide some insight for your question:
One of the most important tests of a forecast - I would argue that it is the single most important one - is called calibration. Out of all the times you said there was a 40% chance of rain, how often did rain actually occur? If, over the long run, it really did rain about 40% of the time, that means your forecasts were well calibrated. If it wound up raining just 20 percent of the time instead, or 60 percent of the time, they weren't.
So this raises a few points. First of all, as you rightly point out, you really can't make any inference about the quality of a single forecast by the result of event which you are forecasting. The best you can do is to see how your model performs over the course of many predictions.
Another thing that is important to think about is that the predictions that Nate Silver provides are not an event itself, but the probability distribution of the event. So in the case of presidential race, he is estimating the probability distribution of Clinton, Trump, or Johnson winning the race. So in this case he is estimating a multinomial distribution.
But he is actually predicting the race at a far more granular level. His predictions estimate the probability distributions of the percentage of votes each candidate will garner in each state. So if we consider 3 candidates, this might be characterized by a random vector of length 51 * 3 and taking values in the interval [0, 1], subject to the constraint that the proportions sum to 1 for the proportions within a state. The number 51 is because other are 50 states + D.C. (and in fact I think it's actually a few more because some states can split their electoral college votes), and the number 3 is due to the number of candidates.
Now you don't have very much data to evaluate his predictions with - he's only provided predictions for the last 3 elections that I'm aware of (was there more?). So I don't think that there is any way to fairly evaluate his model, unless you actually had the model in hand and could evaluate it using simulated data. But there are still some interesting things that you could look at. For example, I think it would be interesting to look at how accurately he predicted the state-by-state voting proportions at a particular time point, e.g. a week out from the election. If you repeat this for multiple time points, e.g. a week out, a month out, 6 months out, and a year out, then you could provide some pretty interesting exposition for his predictions. One important caveat: the results are highly correlated across states within an election so you can't really say that you have 51 states * 3 elections independent prediction instances (i.e. if the model underestimates candidates performance in one state, it will tend to underestimate in other states also). But maybe I would think of it like this anyway just so that you have enough data to do anything meaningful with.