I'm working on a project where the 'in the wild' prevalence is a significant imbalance (e.g. minority 4%). However, I was able to collect data in a balanced manner, i.e. 4,000 samples of minority and 4000 samples of majority. The question is whether or not to use the ROC AUC or prAUC for a measure of skill? I'm training on balanced data but know it will be deployed on imbalanced data, so is it appropriate to extend this logic to my project? I think it is but the more I read the more in the rabbit hole I get...
This cross validated post has a good explanation of when to prioritize AUC-ROC vs AUC-PR. From your post, it sounds like there is a known, imbalanced background prevalence and you are more interested in identifying the rare, positive cases. The precision recall curve should be used as plotting the precision recall curve helps get an idea of the probability the prediction is correct given a positive test result (see more about precision, also known as positive predictive value here).
Before you begin fitting and evaluating models though, I think you should reconsider training on a balanced dataset when there is such a large class imbalance in the real world. Using a balanced dataset throws away value information because the trained model will now believe that both outcomes are equally likely. When the model is applied to new, real world predictions where the outcome is rare, the model isn't likely to perform very well because it's been calibrated to expect equal prevalence among outcomes.
Norm Matloff has a good tutorial on fitting models with imbalanced outcomes. In summary, he suggests maintaining the outcome prevalence in the training dataset and then using a threshold to identify suspicious cases that have a high probability of being the rare outcome. Flagged cases can then be further inspected to determine if they are in fact the rare outcome.