Assume that I have a binary classification problem. The outcome from classification I am mostly interested in is the well-calibrated probabilities.
The first way to check this is the calibration plot (or reliability curve).
The question: is it fair enough to judge about calibration based on Brier score?
Assume that we have "enough" data. Would the classifier with smaller Brier score provide a rather better reliability curve?
My concern comes from the fact that the probability from a classifier are the conditional probabilities. Therefore, I do not see the intuition of applying the Brier score to conditional probabilities.