2
$\begingroup$

I would like to know how we can choose between model fit (calibration) vs AUC when building the predictive model. For example, if I have one predictor which improves the model fit but results in a lower AUC (e.g. 0.80 -> 0.70), would you still include this predictor in this predictive model?

Thank you for the feedback!

$\endgroup$
8
  • $\begingroup$ By what measure does the model fit improve? $\endgroup$
    – Dave
    Nov 28, 2021 at 5:18
  • $\begingroup$ @Dave I am thinking of using the likelihood ratio test to compare models $\endgroup$
    – R Beginner
    Nov 28, 2021 at 18:14
  • $\begingroup$ Likelihood ratio testing considers the performance of the log loss, which is a strictly proper scoring rule. AUC is not strictly proper, so there is a sense in which log loss is preferred to AUC. Frank Harrell, for instance, considers log loss the gold standard for model performance. $\endgroup$
    – Dave
    Nov 28, 2021 at 18:24
  • $\begingroup$ Thank you! When it comes to model fit, between AIC vs likelihood ratio testing, which one is better? $\endgroup$
    – R Beginner
    Nov 28, 2021 at 19:33
  • $\begingroup$ Do you mean AIC or AUC? $\endgroup$
    – Dave
    Nov 28, 2021 at 20:00

1 Answer 1

0
$\begingroup$

Some of the trouble here is that $AUC$ and the likelihood ratio test are based on different ideas.

The $AUC$ measures the extent to which the predictions are separated by true category: the ability of the model to discriminate or discern between categories. Notably, if you divide the predictions by two or apply any other monotonically increasing function (multiplying by $1/2$ is an increasing function), you do not change the order of the predictions, so you do not change the extent to which the predictions are separable into the two categories. Consequently, $AUC$ does not consider the output calibration and if a predicted probability of $p$ corresponds to the event truly happening with probability $p$.

The likelihood involves both the ability of the model to discriminate but also the calibration of the outputs. Consequently, if adding a variable slightly lowers the ability to discriminate but dramatically improves the calibration, the likelihood favors adding this variable. However, the $AUC$ will suffer when you add this variable.

If you want to consider the fit according to the likelihood and also want to have some kind of "absolute" measure of performance (it is hard to say that any particular score counts as "good", but it can be nice to give some context for a likelihood value that lacks an easy interpretation like mean absolute error), you might consider McFadden's $R^2$, which compares the likelihood of your model (fraction numerator) to the likelihood of a reasonable baseline model that always predicts the overall probability (fraction denominator).

$$ R^2_{McFadden} = 1-\left( \dfrac{ \overset{N}{\underset{i=1}{\sum}}\left[ y_i\log(\hat y_i) + (1 - y_i)\log(1 - \hat y_i) \right] }{ \overset{N}{\underset{i=1}{\sum}}\left[ y_i\log(\bar y) + (1 - y_i)\log(1 - \bar y) \right] } \right) $$

In the equation above, $y_i\in\left\{0, 1\right\}$ are the true labels, $\hat y_i$ are the predicted probabilities, and $\bar y$ is the overall probability of the event coded as $1$.

While McFadden's $R^2$ does not seem to be as popular in machine learning circles as the $AUC$, it is part of the literature, and a big part of me thinks that it should be a more popular metric.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.