2
$\begingroup$

I'm trying to fit multiple Stochastic Gradient Descent models to a dataset where the target (binary target, 0 or 1) is very imbalanced, i.e the success rate is about 0.0001.

Out of all the models I've trained, I would like to select the best model based on the validation log-loss and validation AUC. Unfortunately, the average values of the test log-loss (0.001) and the test AUC (0.99) don't allow me to differentiate the models (as all the values are almost the same).

Are these metrics (AUC and LogLoss) good performance metrics for a highly imbalanced classification task? What metrics would allow me to differentiate the models and choose the best one?

Thanks

Improve this question
$\endgroup$
7
  • $\begingroup$ How is the model going to be used? This is the biggest question that should influence which metric you should use to select your model $\endgroup$
    – TBSRounder
    Commented Jun 9, 2017 at 17:50
  • $\begingroup$ After training the model, I'll be using the predicted probabilities. I won't be using 0s or 1s. Does it answer your question? $\endgroup$
    – Aymen
    Commented Jun 9, 2017 at 17:54
  • 1
    $\begingroup$ Log-loss measures how well the probabilities reflect the data they were trained on, so if you're going to be using the probabilities directly, that's the way to go. You should expect that the values of log loss only differ slightly on an absolute scale, as you are attempting to predict a very rare event, meaning all your predicted probabilities would (and should) be small. This isn't a problem for comparing models, but you will want to bootstrap to make sure any differences you observe are consistent across bootstrap samples. $\endgroup$
    – Matthew Drury
    Commented Jun 9, 2017 at 17:55
  • $\begingroup$ Thanks. What do you mean by bootstrap? Cross-validated on multiple (different) test sets? $\endgroup$
    – Aymen
    Commented Jun 9, 2017 at 17:57
  • 1
    $\begingroup$ Yah, log losses don't really make sense on an absolute scale, they are for comparing different models of the same process. $\endgroup$
    – Matthew Drury
    Commented Jun 9, 2017 at 17:58

2 Answers 2

Reset to default
2
$\begingroup$

I think the best way to see performance of the classification with highly imbalanced classes is look at precision-recall curve. You can also use area under this curve as metric.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ In fact, better than the precision-recall curve is the precision-recall gain (PRG) curve; see here for details and implementations. $\endgroup$
    – darXider
    Commented Jun 9, 2017 at 19:49
0
$\begingroup$

The beginning of an answer.

There are some usual metrics for imbalanced data sets. Some of them

Unfortunately I do not know of any intuitions on why to choose one or other metrics.

Improve this answer
$\endgroup$
1
  • 2
    $\begingroup$ It would be more useful to explain how these metrics are affected (or not) by class imbalance. $\endgroup$
    – Black Milk
    Commented Nov 5, 2019 at 18:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.