2
$\begingroup$

I am using gbm(R's caret packages - using train function) on a class imbalanced data set with weights. So, class-1 has a weight of 1 and class-0 has a weight of 10. I am using parameter tuning and minimising AUC. I want to ask that is you are using weights in gbm with a class imbalanced data set then you are atificially making the classifier to put more focus towards the minority class and AUC/ROC is used mainly to check the sensitity & specificity trade-off. Does it make sense to minimise AUC with weights in GBM? or it should be accuracy? Please ignore my lack of understanding.

Thanks.

$\endgroup$
6
  • 1
    $\begingroup$ Auc makes sense for any model, regardless of how it's built as long as it's categorical. It's just a criteria for trade off between false pos and true pos. $\endgroup$
    – meh
    Sep 19, 2015 at 16:57
  • $\begingroup$ AUC is a poor choice of metric for class imbalanced data. Precision-recall works better than sensitivity-specificity. I'm not sure how the weighting will affect this though, but probably just in how you will weight the importance of precision vs recall for your use case $\endgroup$
    – Dan
    Jul 31, 2018 at 15:24
  • $\begingroup$ @Dan Sensitivity to class imbalance is an important distinction between PR and ROC curves, but I think that your explanation would greatly benefit from explaining why sensitivity to class imbalance is important. In my view, class imbalance is mostly an accident of data collection and data availability, so I prefer a measurement which isn't sensitive to that. $\endgroup$
    – Sycorax
    Jul 31, 2018 at 15:45
  • $\begingroup$ @Sycorax heavy class imbalance is surely most often caused by the underlying distribution of what is being measured, not noise to be ignored. Consider a dataset of transactions, we want to classify fraudulent transactions. It may be that 99.999% of transactions are not fraudulent. What does that have to do with data collection or availability. It's simply a property of the problem being solved. $\endgroup$
    – Dan
    Jul 31, 2018 at 16:01
  • $\begingroup$ You're probably right about fraud modeling. I don't know, I've never worked on that problem. But do you believe it's possible that there exists a problem where class imbalance is accidental? If so, that's a reason to prefer ROC analysis. That's all I'm saying -- use the tool that is appropriate to your task. $\endgroup$
    – Sycorax
    Jul 31, 2018 at 16:12

1 Answer 1

3
$\begingroup$

You can use ROC AUC to measure the performance of any binary classifier. There are also extensions of ROC analysis to multi-class classification.

But note that you want to maximize AUC: higher is better.

$\endgroup$
9
  • $\begingroup$ ROC AUC is not great for class imbalanced data: kaggle.com/general/7517 $\endgroup$
    – Dan
    Jul 31, 2018 at 15:27
  • $\begingroup$ You've misread the post. "A large number change in the number of false positives can lead to a small change in the false positive rate used in ROC analysis." Class imbalance doesn't influence FPR. $\endgroup$
    – Sycorax
    Jul 31, 2018 at 15:30
  • $\begingroup$ That's from the question... rather take a look at the answer. Or the paper. It affects how pronounced the difference between the numbers will be when comparing two classifiers. With heavily imbalanced classes, the ROC AUC can be extremely similar for two classifier with very different PR AUC. $\endgroup$
    – Dan
    Jul 31, 2018 at 15:33
  • $\begingroup$ @Dan It seems like you're saying the same thing with different implicit assumptions. I'm saying that a ROC curve is not sensitive to imbalanced classes, and I think that's a good thing. You seem to be saying that you would prefer to use a metric that is sensitive to class imbalance. $\endgroup$
    – Sycorax
    Jul 31, 2018 at 15:38
  • $\begingroup$ All I'm saying is that for this porblem, with imbalanced classes, there are many articles out there suggesting you should rather use PR AUC: stats.stackexchange.com/a/90783/40604 or stats.stackexchange.com/a/90783/40604 or that paper from the link pages.cs.wisc.edu/~jdavis/davisgoadrichcamera2.pdf $\endgroup$
    – Dan
    Jul 31, 2018 at 15:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.