What is the effect of training a model on an imbalanced dataset & using it on a balanced dataset?

Question

When evaluating a model, for example a binary classifier, should the train and test set have 50% + and 50% - label distribution or could the distribution be random?

If the distribution is biased in the train/test sets e.g., 80% + and 20%-, the precision/recall scores may not be representative. For example, the model may do well on classifying positive points but may misclassify a lot of negative points. It's recall is high but its precision could still be high because there aren't too many false positives because there are less negative points in the dataset.

Is AUC robust metric against such imbalanced distributions? Or is it best to balance the distribution in train/test data in order to compute more accurate precision and recall values?

I read this Kaggle forum post: Precision-recall AUC vs ROC AUC for class imbalance problems, but it doesn't discuss the issue I'm raising about dataset distribution.

Answer 1

It is important to understand that improper accuracy scoring rules lead to bogus models and are affected by imbalance (the AUROC (c-index) being one improper scoring rule that is an exception; the c-index is independent of outcome prevalence but is not sensitive enough to be used as an optimality criterion). Use of full probability estimators and proper scoring rules are recipes for success, and handle extreme imbalance. This issues are expanded upon here and here.

Answer 2

If you modify the prevalence of your training set (thereby the prevalence of your CV partitions for model evaluation and selection) you essentially change how the model fits your data (e.g. internally minimizes the error of the fit). This could both cause be beneficial for your goal, or be degradation - that depends on the details of your problem and goal. If you do so, the one important thing is to not modify the prevalence of your held-back test set.

To give an example: if there are unbalanced classes and the model fits data in a way that the positive, less prominent class is badly predicted (bad TPR but good TNR), and you can't change this using hyperparameters, this could likely still be changed e.g. by down- or upsampling, weighting of samples, etc. The internal change is that thereby the less represented samples each contribute a bigger part to the error during fitting, hence how the model fits and represents data.

Besides changing how models fit the data, as you pointed out, one could change the error measure, which largely depends on what you want to achieve. This can be beneficial if you are evaluating and selecting one model from multiple candidate models. For example, using TPR and TNR (+ ROC AUC etc.) boils down to optimizing for correctly predicting samples as positive P samples, if they are actually P - as well as negative N samples, if they are actually N. This essentially leaves out class prevalence (e.g. many false positives don't degrade true positives or false negatives). In contrast, precision and recall (+ precision-recall ROC) look at the same ground information differently, as false positives directly influence both recall and precision.

So, bottom line: it depends on what you want to achieve. There are problems/approaches out there that use artificially manipulated training prevalence to change how the model represented training data - e.g. better representing certain classes - which in turn better satisfies predefined goals. But, as pointed out in the beginning, it's important to not change the prevalence of your final test set, because this one reflects the TPR, TNR, precision, recall, and whichever other metric you look at for the final model on data with (hopefully) natural prevalence.