Assume I’m building a binary classification model to detect a disease that according to research is present in 3% of the population. I’m given a balanced dataset to train the model where half of the people have the disease and the other half do not. I set aside 20% of the dataset for testing and 80% for training with cross validation.

Before making predictions on the test set, should I randomly under-sample the test set’s positive class so it matches the 97-3 distribution that exists in real life? I’m worried that using the balanced dataset in train and test could make performance look better than it is if I’m using precision and recall to evaluate the model. At the same time this would make it harder to detect over- or under-fitting if the distribution is different in the test set. I also don’t want to under-sample the entire dataset because there’s valuable information in all of those positive observations.

Should the train and test set label distribution match what I expect to see when the model is making predictions in a live environment?


1 Answer 1


Undersampling ( or over-...) is rarely a good idea! as you have realized. You should better treat your problem as one of probability estimation, for example with logistic regression. There is many similar questions, and for good links to more information see Rare Events Logistic Regression (or search this site for Frank Harrell).

Also have a look at proper score functions: Why is accuracy not the best measure for assessing classification models?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.