0
$\begingroup$

I'm working on a project where the 'in the wild' prevalence is a significant imbalance (e.g. minority 4%). However, I was able to collect data in a balanced manner, i.e. 4,000 samples of minority and 4000 samples of majority. The question is whether or not to use the ROC AUC or prAUC for a measure of skill? I'm training on balanced data but know it will be deployed on imbalanced data, so is it appropriate to extend this logic to my project? I think it is but the more I read the more in the rabbit hole I get...

$\endgroup$
1
$\begingroup$

This cross validated post has a good explanation of when to prioritize AUC-ROC vs AUC-PR. From your post, it sounds like there is a known, imbalanced background prevalence and you are more interested in identifying the rare, positive cases. The precision recall curve should be used as plotting the precision recall curve helps get an idea of the probability the prediction is correct given a positive test result (see more about precision, also known as positive predictive value here).

Before you begin fitting and evaluating models though, I think you should reconsider training on a balanced dataset when there is such a large class imbalance in the real world. Using a balanced dataset throws away value information because the trained model will now believe that both outcomes are equally likely. When the model is applied to new, real world predictions where the outcome is rare, the model isn't likely to perform very well because it's been calibrated to expect equal prevalence among outcomes.

Norm Matloff has a good tutorial on fitting models with imbalanced outcomes. In summary, he suggests maintaining the outcome prevalence in the training dataset and then using a threshold to identify suspicious cases that have a high probability of being the rare outcome. Flagged cases can then be further inspected to determine if they are in fact the rare outcome.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ I am a simple Stats person; I see Norm's name, I upvote. $\endgroup$ – usεr11852 Mar 16 at 11:32
  • $\begingroup$ Thanks this helps a lot! Thats the road i was going down at first... So my compromise was to test the final models on different prevalance rates and then select the model with the most stable predictions. However, it might make more sense to use 'natural` prevalance rates during training. Im exploring the formula Norm has as well. $\endgroup$ – Josh Erickson Mar 16 at 13:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.