12
$\begingroup$

I am performing a binary classification task where the outcome probability is fair low (aroung 3%). I am trying to decide whether to optimize by AUC or log-loss. As much as I have understood, AUC maximizes the model's ability to discriminate between classes whilst the logloss penalizes the divergency between actual and estimated probabilities. In my task is extremely important to calibrate the precision accuracy. So I would choose logloss, but I wonder whether the best log-loss model should also be the best AUC / GINI models.

$\endgroup$
12
$\begingroup$

As you mention, AUC is a rank statistic (i.e. scale invariant) & log loss is a calibration statistic. One may trivially construct a model which has the same AUC but fails to minimize log loss w.r.t. some other model by scaling the predicted values. Consider:

auc <-  function(prediction, actual) {
  mann_whit <- wilcox.test(prediction~actual)$statistic
  1 - mann_whit / (sum(actual)*as.double(sum(!actual)))
}

log_loss <- function (prediction, actual) {
  -1/length(prediction) * sum(actual * log(prediction) + (1-actual) * log(1-prediction))
}

sampled_data <- function(effect_size, positive_prior = .03, n_obs = 5e3) {
  y <- rbinom(n_obs, size = 1, prob = positive_prior)
  data.frame( y = y,
              x1 =rnorm(n_obs, mean = ifelse(y==1, effect_size, 0)))
}

train_data <- sampled_data(4)
m1 <- glm(y~x1, data = train_data, family = 'binomial')
m2 <- m1
m2$coefficients[2] <- 2 * m2$coefficients[2]

m1_predictions <- predict(m1, newdata = train_data, type= 'response')
m2_predictions <- predict(m2, newdata = train_data, type= 'response')

auc(m1_predictions, train_data$y)
#0.9925867 
auc(m2_predictions, train_data$y)
#0.9925867 

log_loss(m1_predictions, train_data$y)
#0.01985058
log_loss(m2_predictions, train_data$y)
#0.2355433

So, we cannot say that a model maximizing AUC means minimized log loss. Whether a model minimizing log loss corresponds to maximized AUC will rely heavily on the context; class separability, model bias, etc. In practice, one might consider a weak relationship, but in general they are simply different objectives. Consider the following example which grows the class separability (effect size of our predictor):

for (effect_size in 1:7) {
  results <- dplyr::bind_rows(lapply(1:100, function(trial) {
                                    train_data <- sampled_data(effect_size)
                                    m <- glm(y~x1, data = train_data, family = 'binomial')
                                    predictions <- predict(m, type = 'response')
                                    list(auc = auc(predictions, train_data$y),
                                         log_loss = log_loss(predictions, train_data$y),
                                         effect_size = effect_size)
                                  }))
  plot(results$auc, results$log_loss, main = paste("Effect size =", effect_size))
  readline()
}

effect_1

enter image description here

$\endgroup$
  • $\begingroup$ Very informative answer. In you are answer there are two predictions, with same AUC but very different log loss. So I come to this question : I have trained a model for optimizing AUC. But later I realized that I need to go for log loss. I cant afford to retrain the model for log loss (which should be ideal case). Can I apply any transformation on the predictions, so that it has the best log loss. (The log loss considred here is binary meaning, reference probability is either 1 or 0). $\endgroup$ – Rajesh Dachiraju Dec 7 '17 at 6:20
  • 1
    $\begingroup$ What does your model estimate? Log loss only makes sense if you're producing posterior probabilities, which is unlikely for an AUC optimized model. Rank statistics like AUC only consider relative ordering of predictions, so the magnitude of gaps between predictions is irrelevant; clearly this is not the case for probabilities. Any scaling you perform on your AUC optimized predictions will have to address this issue. Furthermore, this only addresses calibration of your predictions towards a reasonable posterior estimate, not global minimization of LL, as outlined in this post. $\endgroup$ – khol Dec 7 '17 at 7:15
  • 1
    $\begingroup$ You might be interested in platt scaling as a starting point. $\endgroup$ – khol Dec 7 '17 at 7:16
  • $\begingroup$ I understand its not global. I just want to setup a transformation which might have a parameter and then play around with it. $\endgroup$ – Rajesh Dachiraju Dec 7 '17 at 7:33
0
$\begingroup$

For imbalanced labels, area under precision-recall curve is preferable to AUC (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4349800/ or python scikit-learn docs)

Also, if your goal is to maximize precision, you can consider doing cross-validation to select the best model (algorithm + hyperparameters) using "precision" as the performance metric.

$\endgroup$
  • 2
    $\begingroup$ I downvoted for a few reasons here. 1) You should cite either a source or a reason for your first statement. 2) How exactly do you optimize for precision? Wouldn't that create a degenerate model? $\endgroup$ – Matthew Drury Feb 26 at 19:18
  • 1
    $\begingroup$ ok thanks for helping me make a better answer. I added 2 references. and how do you optimize for precision? just like any other metric. You just specify "precision" as your scoring function, for example in scikit-learn. Precision is a metric like accuracy, AUC, etc $\endgroup$ – Paul Feb 26 at 19:43
  • $\begingroup$ Ahh, sure. But I think when people read "optimize" they are going to assume it's during the training of your model. I think sklearn gets this pretty wrong, since it uses a fixed classification threshold, and you should really be tuning that with cross validation. $\endgroup$ – Matthew Drury Feb 26 at 20:18
  • $\begingroup$ Yep, I now see how "optimize" may be confusing. Not sure how OP meant it. For me, it's like you say: tune hyperparameters via CV so precision is maximized. And I think that's how you apply it in sklearn as well. $\endgroup$ – Paul Feb 27 at 6:17
  • $\begingroup$ Sure, I think your answer would be improved here if you added that clarification. A bit off topic, but I actually think sklearn is quite poor at supporting that, since it tunes based on a fixed classification threshold, which I would argue is quite bad practice. $\endgroup$ – Matthew Drury Feb 27 at 16:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.