A requirement for many statistical and machine learning prediction models is the independence of observations. However, when using oversampling we reintroduce the same observations in the dataset. How can this be legitimate given the independence requirement?


1 Answer 1


Indeed, oversampling leads the duplicate samples and therefore the samples are no longer independent. Other than that, the distribution on the oversampled data set no longer represents the natural distribution from which the samples were taken and on which the classifier will be evaluated.

If so, one might ask why to use oversampling at all? The reason to the usage is that in many times "the blanket is too small". We would like to have a data set that represents the natural distribution, balanced and with enough samples for each class.

When the natural distribution is imbalanced itself, we obviously cannot both represent the natural distribution and have a balanced data set.

However, building data set with independent samples or certain representation requirements should serve us. When we cannot obey these requirement, or when we will benefit more from not obeying them, it is OK to manipulate the data. For example, the entire field of boosting is based on manipulating the distributions provided to the learners.

What one should remember that these requirement usually protect us. If the samples are not independent, you cannot compute probabilities in a straight forward way. If you trained a model on one distribution, you are not sure how it will perform on others.

If you choose to violate such requirement, make sure to have different safety measure that will catch you if you'll go wrong. For example, manipulate the data set with over sampling but test on a independent dat set set that represents your natural distribution.

By the way, I don't like oversampling due to the reason above. There are better ways to cope with imbalanced data sets. For details see here.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.