Base rate of accuracy after resampling for classification problems

Question

If I had an imbalanced dataset with 10% positive instances and 90% negative ones, the base rate for accuracy before resampling is 90%. But what about I resampled the data such that I have an equal amount of positive and negative instances? Will 50% be my new base rate for accuracy?

I am asking this question because I found that, after resampling, my machine learning model's accuracy dropped but precision, recall, and FPR all improved on the validation set.

On a related note, will resampling techniques generally reduce the accuracy but improve precision and recall?

Thanks very much!

Answer 1

Using resampling to balance the training set such that it has a 50:50 split will indeed change the base rate accuracy on the training set to be 50:50. That said, this is not relevant for us because we should be using the separate test-set to assess the performance of our algorithm. So yes, 50% will be the new base rate accuracy on the training set but for our test set the base rate remains 90%. Even if we artificially balance our training set for some reason, we should definitely avoid doing this step in our test-set.

Changes in the metrics reported (especially the higher recall) are expected; our algorithm is presented with more instances of the minority class, it will therefore be more likely to optimise itself to recognise minority class instances. So yes, this resampling step will have direct impact to the overall metrics observed. Usually Recall increases simply by virtue of having more samples of the minority class and/or decreasing the overlap between classes on the sample sample. Getting higher Precision is a bit more of happy coincidence here really; probably some noisy samples of the majority class where excluded but that is not generally the case.

The above being said I would note two things:

1. using Precision-Recall and especially Accuracy is not a good tactic to pick the best performing classifier unless there is a particular requirement to do so. (e.g. we need to have at least 80% Recall). There is a great thread on the matter on CV.SE regarding "Why is accuracy not the best measure for assessing classification models?". I would suggest using another method to assess classifier performance; some quick examples would be: ROC-AUC (Receiver operating characteristic - Area under the curve; very popular and widely used choice), PR-AUC (Precision Recall - Area under the curve; popular choice for imbalanced learning models) or Brier score (a proper scoring rule when it comes to classifier performance).
2. a 90:10 split between the majority and the minority class is usually not very problematic unless the sample is small and/or relative uninformative. Using a better way to assess classifier performance is more relevant than immediately resorting to under/over-sampling our training data, using cost-aware techniques and other specialised techniques for imbalanced learning.