Linked Questions

56
votes
6answers
30k views

Binary classification with strongly unbalanced classes

I have a data set in the form of (features, binary output 0 or 1), but 1 happens pretty rarely, so just by always predicting 0, I get accuracy between 70% and 90% (depending on the particular data I ...
20
votes
3answers
9k views

ROC vs Precision-recall curves on imbalanced dataset

I just finished reading this discussion. They argue that PR AUC is better than ROC AUC on imbalanced dataset. For example, we have 10 samples in test dataset. 9 samples are positive and 1 is ...
20
votes
1answer
5k views

Choosing between loss functions for binary classification

I work in a problem domain where people often report ROC-AUC or AveP (average precision). However, I recently found papers that optimize Log Loss instead, while yet others report Hinge Loss. While I ...
19
votes
2answers
8k views

What is “baseline” in precision recall curve

I'm trying to understand precision recall curve, I understand what precision and recall are but the thing I don't understand is the "baseline" value. I was reading this link https://classeval....
11
votes
3answers
8k views

“Good” classifier destroyed my Precision-Recall curve. What happened?

I'm working with imbalanced data, where there are about 40 class=0 cases for every class=1. I can reasonably discriminate between the classes using individual features, and training a naive Bayes and ...
7
votes
1answer
2k views

outlier detection: area under precision recall curve

I would like to compare outlier detection algorithms. I am not sure if area under roc or under precision recall curve is the measure to use. A quick test in matlab gives me strange results. I try to ...
4
votes
3answers
6k views

How can I calculate the false positive rate for an object detection algorithm, where I can have multiple objects per image?

I have an object detection algorithm for which I would like to plot an ROC curve. For this, I need the values of the fall-out corresponding to values of recall. The false positive rate, or fall-out, ...
1
vote
1answer
110 views

performance measure suited for imbalanced classes and robust towards changing class ratios

I am looking for the best performance measure. My use case: I want to find out which dataset can be modelled best with binary classification. The datasets have an active minority class I am ...
1
vote
0answers
45 views

Will oversampling help with generalization (small imbalanced dataset)?

I have an imbalanced dataset (2:1 ratio) with about 60 patients and 80 features. I performed RFE + stratified cross validation to reduce the features to 15 and I get an AUC of 0.9 with Logistic ...
1
vote
0answers
44 views

PR AUC vs ROC AUC for imbalanced data [duplicate]

I am struggling with choosing metric that I will use to compare models performance and hiperparam search. My task is similar to fraud detection. I have found out that many people states PR is better ...
0
votes
1answer
41 views

precision recall AUC or ROC AUC question

I'm working on a project where the 'in the wild' prevalence is a significant imbalance (e.g. minority 4%). However, I was able to collect data in a balanced manner, i.e. 4,000 samples of minority and ...
0
votes
1answer
203 views

Imbalanced data-set : rare class v.s. rare events

I am currently working on an imbalanced data-set (1% of 1). However I am a bit concerned by the underlying model. I treated the problem as a classification problem, making some hypotheses on the ...