Assume I’m building a binary classification model to detect a disease that according to research is present in 3% of the population. I’m given a balanced dataset to train the model where half of the people have the disease and the other half do not. I set aside 20% of the dataset for testing and 80% for training with cross validation.
Before making predictions on the test set, should I randomly under-sample the test set’s positive class so it matches the 97-3 distribution that exists in real life? I’m worried that using the balanced dataset in train and test could make performance look better than it is if I’m using precision and recall to evaluate the model. At the same time this would make it harder to detect over- or under-fitting if the distribution is different in the test set. I also don’t want to under-sample the entire dataset because there’s valuable information in all of those positive observations.
Should the train and test set label distribution match what I expect to see when the model is making predictions in a live environment?