probability calibration and Brier score Assume that I have a binary classification problem. The outcome from classification I am mostly interested in is the well-calibrated probabilities.
The first way to check this is the calibration plot (or reliability curve).
The question: is it fair enough to judge about calibration based on Brier score?
Assume that we have "enough" data. Would the classifier with smaller Brier score provide a rather better reliability curve?
My concern comes from the fact that the probability from a classifier are the conditional probabilities. Therefore, I do not see the intuition of applying the Brier score to conditional probabilities.
 A: The short answer is that it only makes sense to calculate the Brier score for the conditional probabilities, $\hat y = P(y=1|X)$, where $y$ is the outcome, $\hat y$ is your prediction, and $X$ are your predictors.
In other words, $\hat y$ is the probability that $y=1$, conditional on this particular value of the predictors, $X$.
The Brier score in this case is just
$$
\frac{1}{N}\sum_i^N (\hat y_i - y_i)^2 
$$

What other kinds of probability could there be?
The only other option here is the marginal probability, $P(y=1)$. We can estimate this by simply counting the proportion of times $y=1$ in the data. Clearly, it doesn't make sense to use this value when calculating the Brier score!


Would the classifier with smaller Brier score provide a rather better reliability curve?

Yes. If your classifier predicts $\hat y = 1$ in all cases where $y = 1$ and $\hat y = 0$ where $y = 0$, it has a Brier score of $0$. If it does the opposite, it has a score of $\pm1^2 = 1$. In most cases, such perfect predictions won't be possible, but a good classifier can still be well calibrated, for instance by predicting $\hat y = 0.5$ in cases where $y = 1$ half the time and $y=0$ the rest. A classifier that does this will have the lowest possible Brier score on your data.
