1
$\begingroup$

I understand that the PR-AUC provides a better accuracy estimate than the ROC-AUC in the case of highly skewed datasets. But if I have a test dataset with less than 5 positives and 10000+ negatives, can I still use it? Is there a better alternative in such cases?

Buy the way, I have calculated the ROC-AUC based on the test dataset above and I got AUC>0.75. I think this is probably an overly optimistic assessment of the classififers's performance and, hence, I am looking for a better peformance measure.

$\endgroup$
2
$\begingroup$

Your effective sample size is 5 so you do not have an adequate sample for doing anything, not even predicting the overall average outcome in the absence of any predictors. So the choices you make about measuring accuracy are "too late". Note that when $Y$ is binary it is not a good idea to consider classification. Better almost always is to predict the probability that $Y=1$. Classification is an arbitrary manipulation that has to be completely re-thought when you move to a sample that has a prevalence of $Y=1$ that is much different than $\frac{5}{10000}$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.