This question already has an answer here:
Let us suppose there is this pollster that over many years had given predictions on bilateral races. The prediction is always in the form:
"In the race between A and B I project that A wins with a XX% confidence".
A, B, whos wins and the confidence XX% are different each time. But the format is the same.
The question: How can we give a score to this pollster telling us not just how accurate his prediction are, but also how accurate is his confidence marks?
If XX% is the same number always we can measure the actual accuracy (by waiting for the real race results) and compare the pollster's confidence to the actual accuracy. Good match between these numbers will mean that the pollster, while might not being always correct, can at least estimate his confidence well.
However, what is the procedure if XX% is different each time?
