Suppose a binary outcome $Y=0$ or $Y=1$ where $P(Y=1|X)=f(X)$ is a function of $X$. The goal is to estimate $f$ as closely as possible using a classifier that returns a probability estimate (e.g. random forest or regularized logistic regression) where the tuning parameter is chosen using cross-validation. Which measure of model fit should be applied in the validation sets?

Clearly, $P(Y=1|X)$ is not observed (only $Y$ is) but the objective is estimating $P(Y=1|X)$ properly and not (only) classifying $Y$ accurately. These appear two different tasks. For example, a classifier always predicting 0.51 whenever $P(Y=1|X)>0.5$ has good classification performance with cut-off at 0.5, but does not necessarily predict the probability well.

Which measure of fit should one choose given this objective? Options I know of are accuracy defined as the proportion of validation $Y$ classified to the correct class and $AUC$, area under the receiver operating characteristic curve. But I do not know abou their performance for my objective.


Binary cross-entropy is probably the most commonly used loss function for binary classification. The loss function takes the form

$$-\sum_i \left(y_i\times \ln(p_i) + (1-y_i)\times \ln(1 - p_i)\right),$$

where $y_i$ is the $i$th observation of the true output and $p_i$ is the predicted probability that $Y_i=1$. It can be seen that in the case where $y_i=1$, losses are lower for $p_i$ close to 1 and larger for $p_i$ close to 0; the opposite holds when $y_i=0$.

Binary cross-entropy is frequently preferred to cutoff loss functions like you describe in your question because it is differentiable everywhere (for $p_i\in(0, 1)$), so that standard optimization methods can be used.

| cite | improve this answer | |
  • $\begingroup$ Following this argument one may also use the squared error loss $n^{-1}\sum( y_i - \hat{p}_i )^2$ (MSE / Brier score). But do these measures have proven performance to estimate true $p$ and which one is better? $\endgroup$ – tomka Jun 7 '19 at 10:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.