Which cross-validation measure of model fit performs best when the objective is probability estimation in classification tasks?

Question

Suppose a binary outcome $Y=0$ or $Y=1$ where $P(Y=1|X)=f(X)$ is a function of $X$. The goal is to estimate $f$ as closely as possible using a classifier that returns a probability estimate (e.g. random forest or regularized logistic regression) where the tuning parameter is chosen using cross-validation. Which measure of model fit should be applied in the validation sets?

Clearly, $P(Y=1|X)$ is not observed (only $Y$ is) but the objective is estimating $P(Y=1|X)$ properly and not (only) classifying $Y$ accurately. These appear two different tasks. For example, a classifier always predicting 0.51 whenever $P(Y=1|X)>0.5$ has good classification performance with cut-off at 0.5, but does not necessarily predict the probability well.

Which measure of fit should one choose given this objective? Options I know of are accuracy defined as the proportion of validation $Y$ classified to the correct class and $AUC$, area under the receiver operating characteristic curve. But I do not know abou their performance for my objective.

Answer 1

Binary cross-entropy is probably the most commonly used loss function for binary classification. The loss function takes the form

$$-\sum_i \left(y_i\times \ln(p_i) + (1-y_i)\times \ln(1 - p_i)\right),$$

where $y_i$ is the $i$th observation of the true output and $p_i$ is the predicted probability that $Y_i=1$. It can be seen that in the case where $y_i=1$, losses are lower for $p_i$ close to 1 and larger for $p_i$ close to 0; the opposite holds when $y_i=0$.

Binary cross-entropy is frequently preferred to cutoff loss functions like you describe in your question because it is differentiable everywhere (for $p_i\in(0, 1)$), so that standard optimization methods can be used.