# Which cross-validation measure of model fit performs best when the objective is probability estimation in classification tasks?

Suppose a binary outcome $$Y=0$$ or $$Y=1$$ where $$P(Y=1|X)=f(X)$$ is a function of $$X$$. The goal is to estimate $$f$$ as closely as possible using a classifier that returns a probability estimate (e.g. random forest or regularized logistic regression) where the tuning parameter is chosen using cross-validation. Which measure of model fit should be applied in the validation sets?

Clearly, $$P(Y=1|X)$$ is not observed (only $$Y$$ is) but the objective is estimating $$P(Y=1|X)$$ properly and not (only) classifying $$Y$$ accurately. These appear two different tasks. For example, a classifier always predicting 0.51 whenever $$P(Y=1|X)>0.5$$ has good classification performance with cut-off at 0.5, but does not necessarily predict the probability well.

Which measure of fit should one choose given this objective? Options I know of are accuracy defined as the proportion of validation $$Y$$ classified to the correct class and $$AUC$$, area under the receiver operating characteristic curve. But I do not know abou their performance for my objective.

$$-\sum_i \left(y_i\times \ln(p_i) + (1-y_i)\times \ln(1 - p_i)\right),$$
where $$y_i$$ is the $$i$$th observation of the true output and $$p_i$$ is the predicted probability that $$Y_i=1$$. It can be seen that in the case where $$y_i=1$$, losses are lower for $$p_i$$ close to 1 and larger for $$p_i$$ close to 0; the opposite holds when $$y_i=0$$.
Binary cross-entropy is frequently preferred to cutoff loss functions like you describe in your question because it is differentiable everywhere (for $$p_i\in(0, 1)$$), so that standard optimization methods can be used.
• Following this argument one may also use the squared error loss $n^{-1}\sum( y_i - \hat{p}_i )^2$ (MSE / Brier score). But do these measures have proven performance to estimate true $p$ and which one is better? – tomka Jun 7 '19 at 10:09