11
$\begingroup$

I've trained two models (binary classifiers using h2o AutoML) and I want to select one to use. I have the following results:

 model_id        auc     logloss    logloss_train   logloss_valid   gini_train  gini_valid

DL_grid_1   0.542694    0.287469         0.092717        0.211956     0.872932    0.312975
DL_grid_2   0.543685    0.251431         0.082616        0.186196     0.900955    0.312662

the auc and logloss columns are the cross-validation metrics (the cross validation only uses the training data). the ..._train and ..._valid metrics are found by running the training and validation metrics through the models respectively. I want to either use the logloss_valid or the gini_valid to choose a the best model.

Model 1 has a better gini (i.e. better AUC) but model two has a better logloss. My question is which one to choose which I think begs the question, what are the advantages/disadvantages to using either gini (AUC) or logloss as a decision metric.

$\endgroup$
  • $\begingroup$ This video has a nice explanation of why logloss is preferred if you're interested in the probabilities and not just the classification. Note that for binary classification, logloss is equal to the brier score. $\endgroup$ – Dan Jul 23 at 10:20
9
$\begingroup$

Whereas the AUC is computed with regards to binary classification with a varying decision threshold, logloss actually takes "certainty" of classification into account.

Therefore to my understanding, logloss conceptually goes beyond AUC and is especially relevant in cases with imbalanced data or in case of unequally distributed error cost (for example detection of a deadly disease).

In addition to this very basic answer, you might want to have a look at optimizing auc vs logloss in binary classification problems

A simple example of logloss computation and the underlying concept is discussed in this recent question Log Loss function in scikit-learn returns different values

In addition, a very good point has been made in stackoverflow

One must understand crucial difference between AUC ROC and "point-wise" metrics like accuracy/precision etc. ROC is a function of a threshold. Given a model (classifier) that outputs the probability of belonging to each class we usually classify element to the class with the highest support. However, sometimes we can get better scores by changing this rule and requiring one support to be 2 times bigger than the other to actually classify as a given class. This is often true for imbalanced datasets. This way you are actually modifing the learned prior of classes to better fit your data. ROC looks at "what would happen if I change this threshold to all possible values" and then AUC ROC computes the integral of such a curve.

$\endgroup$
  • $\begingroup$ Your first link contains "AUC maximizes the model's ability to discriminate between classes whilst the logloss penalizes the divergency between actual and estimated probabilities", so I'm basically asking how do I chose between those two objectives? Is it a case of if I work directly with the 'probability' output of the model I should optimise logloss (and then probably also do a probability calibration) where-as if I'm actually using it as a classifier to make hard decision that it is one calss or the other I should go with AUC? $\endgroup$ – Dan Jan 10 '18 at 14:55
  • 1
    $\begingroup$ If you just care about a right or wrong decision --> use AUC - if you care about how much "right" or "wrong" you are --> use logloss $\endgroup$ – Nikolas Rieble Jan 11 '18 at 8:59
  • $\begingroup$ Any further choice of performance measure can not be generalised but rather depends on the domain / application. $\endgroup$ – Nikolas Rieble Jan 11 '18 at 9:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.