I'm modelling a rare event (say 1 in 10000) and I'm using an over sampled train set to cross validate and train my model. I'm using ROC as a global performance metric but there are business reasons for which I want to be able to communicate the false positive at a certain cutoff specified by our business folks (say 0.95). Based on my over sampled train set my model produces an ROC of 0.97 (5 folds cross validation). If I look at the curve I can determine that the False Positive rate at cutoff 0.95 is 0.0003. Give that, when I score a test set and take all observations with a probability of 0.95 or greater, can I state that on average 0.0003 of those will be false positives? Or do I need to adjust 0.0003 to account for the true class distribution of my rare event?
Given the strongly imbalanced dataset, avoid oversampling your dataset and focus on predicting correct probabilities.
In more detail: As with any classification task, we have to consider our utility function in terms of correct outcomes and choose the action that optimises the expected utility. More specifically, because we are dealing with a imbalanced dataset, using the AUC-ROC only is probably a bit short-sighted. Using AUC-RP and/or calibration curves to showcase the performance of the classifier is more relevant. A best case scenario would be using a proper scoring rule like the logarithmic scoring rule or Brier score so we convey a coherent idea of the classifier's performance. In that respect because you work with "business folks" try reporting to the MAE of the probability predictions. MAE is an improper scoring rule for probabilistic outcomes but pretty much everyone can relate to an "expected absolute error" irrespective of the Maths background (the "squared error" is hard to explain to people who do not know the difference between variance and standard deviation already).
I would also recommend caution when it comes to oversampling the available data. This again can be relevant in some cases but realistically it changes the underlying distribution of our data. Given our model is well-calibrated (check the respective calibration curve), our classifier has no problem being relevant to another (unseen) sample of interest. In contrast the insights from a classifier trained on an over-/under-sampled training sample do not transfer to the general population and we have little control as to why the classifier might not generalise correctly as the new unseen sample of interest will have by definition a very different distribution for features to our (over-sampled) training set.