What are the differences and common points, if any, between oversampling as a survey design method and oversampling in a machine learning context?

Question

I've seen the term "oversampling" used in a survey design methodology context and in a machine learning context (e.g. methods like SMOTE). I'm intrigued by the differences between the two.

So far, here's what I understood:

The purpose of oversampling in survey design is to reduce the variance for a target sub-population with a rare but interesting feature, and to cut costs related to sampling the population, as explained here: https://ajph.aphapublications.org/doi/full/10.2105/AJPH.2017.303895 Oversampling is planned beforehand, and is done when we collect the data. Weights are then applied to observations to account for the bias introduced by oversampling.

In machine learning, the goal of oversampling is to improve prediction metrics (e.g. accuracy) for so-called imbalanced datasets. Oversampling is done when the data has already been collected, so it involves duplicating observations or creating synthetic observations. In this context, oversampling has been criticized as a solution to a non-problem (as many posts on this website explain well). For example accuracy may be simply the wrong indicator to look at, and other methods than oversampling may be applied to improve predictions.

Am I correct as to the differences between the two contexts? Are there other major differences (or common points) I am missing?

Bonus question: has oversampling in machine learning been originally inspired by oversampling as a survey design method? If so, why does it not use weights in the same way as in survey design?

I don't have any particular practical problem related to any of this, my question is just out of curiosity, as it's (mildly) surprising to have the same term used in two apparently quite different contexts. So I'm afraid I cannot be really more specific than what I ask above. Thanks!

Answer 1

You are correct about oversampling in survey design. If you want to read more about it, another useful search terms is "stratified sampling" (which is itself another term that survey stats and ML use in different ways!).

To the best of my knowledge you are right about oversampling in ML too. However, others may be able to answer that better than I can.

Finally, I don't know the history of why oversampling in ML came to be used the way it is, but I doubt that it came from survey statistics. If the people who introduced oversampling to ML had a background in statistical survey methods, they probably would have introduced sampling weights instead. Their background would have primed them to realize that ML-style oversampling effectively makes you think you have lower standard errors than you really do.

Instead, I suspect it was developed independently by ML practitioners with limited statistical background. My guess is that someone thought, "I want to deal with class imbalance somehow, and it's easy to write a few lines of code to oversample the minority class," and they just didn't have the right background to think of the downsides or know of the alternatives. Even when people do realize the risks of oversampling and do think of using weights, it's not always obvious how to incorporate sampling weights sensibly into a new ML algorithm.

(Edit: About that last comment---there are at least 3 common types of weights used in statistics. Thomas Lumley summarizes them as precision weights ("this row in the dataset is actually the sample mean of 10 different units"), frequency weights ("we've compressed our dataset by noting that 10 different units all had identical values on each variable"), and sampling weights ("this unit was sampled from the population with probability 1/10"). The different types of weights often give similar point estimates or model predictions, but not always; and they give very different standard errors. Precision weights and frequency weights seem like a more natural fit for the ML world ("we acquired this large dataset, without using any particular deliberate sampling design, and then we compressed it to take up less disk space"). So when ML algorithms are adapted to use weights at all, I suspect the priority is often on using precision or frequency weights, not sampling weights.)