5
$\begingroup$

I have a dataset that contains 99.95% 0's and 0.05% 1's as the target. The dataset contains million rows. I want to build a binary classification model that predicts almost all the 1's correctly while keeping the false positives at minimum.

I have read it somewhere that AUC-PRC is a better metric for the above scenario compared to AUC-ROC. Is it correct?

$\endgroup$

3 Answers 3

3
$\begingroup$

Neither seems appropriate. Rather, assign whatever penalty scores you want to the two kinds of errors (mistaking a 0 for a 1, and mistaking a 1 for a 0) and sum the errors. This allows you to precisely control the tradeoff.

$\endgroup$
10
  • $\begingroup$ Thank you for your answer. I am new to machine learning and don't know how to implement your answer in a code in R. Could you please elaborate a little? How can i assign penalty score in a model? I plan to use some boosting method to model the data and later modify things based on the results i get. $\endgroup$
    – Aman
    Feb 21, 2017 at 3:24
  • $\begingroup$ I believe Kodo is telling about weighed accuracy $\endgroup$
    – SmallChess
    Feb 21, 2017 at 5:27
  • $\begingroup$ @Aman That sounds like a programming question, in which case Stack Overflow, not here, is the right site to ask. $\endgroup$ Feb 21, 2017 at 15:25
  • 1
    $\begingroup$ I think you are over-simplifying things. While imperfect (a "perfect metric" is very subjective) AUCPR is far from totally inappropriate. See some references on the subject. (eg. see @Marc's answer here and papers like here and here. It is better for people to use AUCPR as they are unlikely to misspecify it than have them put arbitrary weights and penalties ending up with a "salad metric". $\endgroup$
    – usεr11852
    Feb 25, 2017 at 13:26
  • $\begingroup$ @usεr11852 Notice that the OP wants to build a classifier, whereas the two AUC measures attempt to summarize the performance of a signal over a number of classifiers that could be defined with it. Only the classifier actually used matters. $\endgroup$ Feb 25, 2017 at 16:40
2
$\begingroup$

You can look at the Precision,Recall and the F1 score which is nothing but the harmonic mean of the Precision and Recall.

$\endgroup$
2
$\begingroup$

Your reading is correct in the sense that AUC-PRC is a better metric for imbalanced classification compared to AUC-ROC. I disagree with Kodi in sense that AUC could be useful in these scenarios. Like Santanu said you could look for precision, recall and F1. I would want to add Sensitivity and Kappa.

However, choice of a metric is not only the way to handle imbalanced classification. You could look for sampling techniques such as SMOTE, converting it to a probability estimation problem with biased threshold and others discussed here and elsewhere.

$\endgroup$
2
  • 1
    $\begingroup$ Why is AUC-PRC better than AUC-ROC for imbalanced classes. I could not find a good explaination $\endgroup$
    – Aman
    Feb 22, 2017 at 4:56
  • 1
    $\begingroup$ Have you had a look at this: stats.stackexchange.com/a/90783/73547 ? $\endgroup$
    – discipulus
    Feb 22, 2017 at 5:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.