I am doing a research on the reliability of different models in detecting hidden defects in a test specimen. I have made a test specimen with defect prevalence about 25% (12 positive out of 49 total points). Is this data considered as a balanced or imbalanced data for this purpose? What do you recommend to use with this data set, ROC or PR curve? if it is imbalanced, what is the minimum percentile that satisfies the balanced data condition?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
2
There is no hard threshold, but this is considered balanced. If you have 1% or less positives the situation is very different.
The question, however, depends not so much on class balance as on what performance measures you care about. ROC and PR curves, though related, visualize different aspects of a model.
answered Feb 2, 2015 at 7:27
-
$\begingroup$ Thanks Marc Claesen for your response. I am not a statistician, so simply I went with ROC curve since it doesn't depend on the "positive events" prevalence. I need this feature because, in reality, we don't know the actual prevalence of a positive events, and that's why ROC curve is more preferable then PR (at least by me). If I am wrong please correct me. $\endgroup$ Commented Feb 2, 2015 at 18:11
-
$\begingroup$ By the way, would you mind to refer/provide any reference supporting your answer. I've been trying to find something clearly states this, but unfortunately, I couldn't yet. $\endgroup$ Commented Feb 2, 2015 at 18:15