How to see how good my probability estimations are? I am estimating the probability for an event to happen based on certain criteria prior to the event. The event either happens or does not happen (1 or 0). I have a few hundred thousand such pairs of 1) prior estimation in % and 2) 1/0 (happens/does not happen).
My question is: What is the best scientific method to test how good my estimations were? My hypothesis is that I am over/underestimating the probability "in the tails" (e.g. near to 0% or near to 100%). I have tried splitting the pairs into a number of ranges, which I have arbitrarily determined (e.g. 0-10%, 10%-20%, ... etc) to see how close I am in each bracket, but the results do not seem very revealing. I would be thankful if someone told me a better way or at least give me some direction.
 A: Density or probabilistic forecasts can be evaluated using so-called scoring-rules (you may want to look through previous questions with this tag). A scoring rule is simply a function that maps a probabilistic forecast ("60% chance of the event happening, 40% of not") and the corresponding outcome to a value. These scores are then averaged. Crucially, scoring rules can be used for probabilistic forecasts that vary between events that you forecast. The disadvantage is that scoring rules are not overly intuitive, and you can't say what a "good" value is.
An early paper for dichotomous forecasts is Murphy & Winkler, 1987, Monthly Weather Review. More recently, Tilman Gneiting has been working extensively on this. One review is Gneiting & Katzfuss, 2014, Annual Review of Statistics and its Application. Gneiting & Raftery (2007, JASA) give examples for (strictly) proper scoring rules for discrete variables. If you are most interested in specific values of your prediction, then Gneiting & Ranjan, 2011, Journal of Business & Economic Statistics or Diks et al., 2011, Journal of Econometrics may be helpful.
Let's look at an example. Suppose we have a dichotomous variable. We have three different potential predictive densities: our predictive density is one of


*

*either $\widehat{p_0}=1$ and $\widehat{p_1}=0$

*or $\widehat{p_0}=0.5$ and $\widehat{p_1}=0.5$

*or $\widehat{p_0}=0$ and $\widehat{p_1}=1$


Suppose further that the true future density is also one of these. We thus have $3\times 3=9$ potential combinations of predictive and true densities.
We assess predictive densities using the Brier or quadratic score. For an outcome $i$, this is
$$ -2\widehat{p_i} + \sum_{j=0}^1 \widehat{p_j}^2,$$
of which we will look at the expectation. The first two cases are:


*

*The predictive distribution is $\widehat{p_0}=1$ and $\widehat{p_1}=0$, and this is indeed the true distribution. Then the expected Brier score is
$$ -2(1\times1 + 0\times0)+1 = -1.$$

*The predictive distribution is $\widehat{p_0}=\widehat{p_1}=0.5$, but the true distribution is $p_0=1$ and $p_1=0.5$. Then the expected Brier score is
$$ -2(1\times 0.5+0\times 0.5)+0.25+0.25 = -0.5.$$


The full table is:
$$
\begin{array}{c|ccc}
 & p_0=1, p_1=0 & p_0=p_1=0.5 & p_0=0, p_1=1 \\
 \hline
 \widehat{p_0}=1, \widehat{p_1}=0 & -1 & 1 & 1 \\
 \widehat{p_0}=\widehat{p_1}=0.5 & -0.5 & -0.75 & -0.5 \\
 \widehat{p_0}=0, \widehat{p_1}=1 & 1 & 0 & -1
\end{array}
$$
We see how in each column, the correct predictive distribution gives the lowest expected Brier score, in the diagonal cell. This is not overly surprising, since by Gneiting & Raftery (2007, JASA), the Brier score is strictly proper, i.e., it is minimized in expectation by the correct future distribution, and by no other.
