2
$\begingroup$

Cross Validated has a pretty thorough debunking of class imbalance being an inherent problem for SMOTE to solve.

However, SMOTE is explicitly a method for synthesizing new points.

Is SMOTE any good at synthesizing new points? If so, is there any way to use SMOTE-style point synthesis to improve model performance, even if we respect the natural class ratio by synthesizing points from all classes, not just the minority class?

(I wrote “SMOTE-style point synthesis” because the “M” means “minority”, so it’s hard to argue that we’re doing SMOTE if we synthesize new points from the majority class, even if we use the same approach.)

$\endgroup$
4
  • $\begingroup$ I think the first the thing to we assess if is good or not is defining a metric to see the goodness. Maybe see if the new data points don't distort the original distribution, or if the most cases improve the model performance. So one need to come with some tests to evaluate if it holds. $\endgroup$
    – Allan
    Aug 11, 2022 at 14:00
  • $\begingroup$ I asked a question here: stats.stackexchange.com/questions/559294/… requesting examples where algorithms such as SMOTE demonstrably improve accuracy. It wen unanswered, even with a modest bounty. I think that is a partial answer in itself! ;o) $\endgroup$ Aug 11, 2022 at 15:56
  • 1
    $\begingroup$ @DikranMarsupial I remember that question of yours and agree that the lack of an answer is something of an answer itself! However, I see a difference between that question and mine in that yours deals with changing the class ratio, while mine just wants to know if SMOTE is any good at synthesizing observations that are plausible elements of a group. $\endgroup$
    – Dave
    Sep 9, 2022 at 3:23
  • $\begingroup$ In my experience, SMOTE is an idiotic idea which is effectively just an obfuscated way of adding more copies of existing data points to the data set. I have seen it cause plenty of problems, but I've never seen it actually improve a model. $\endgroup$
    – Flounderer
    Sep 9, 2022 at 7:33

2 Answers 2

4
$\begingroup$

No, probably not.

(i) SMOTE can only generate synthetic data points within the convex hull of a subset of points (and therefore will be within the convex hull of the full set of points). In the true distribution, it is likely that there will be points that fall outside this distribution. For example, if you split the data to form test and training sets, some of the test points are likely to lie outside the convex hull of the training set (especially in high dimensions). SMOTE will be fundamentally unable to synthesise points in those positions, and hence cannot be representative of the underlying distribution.

(ii) SMOTE generates points lying on straight lines connecting pairs of existing data points. It is extremely unlikely that the true distribution is structured in that way - for a start it means that the data points you have been given are "special" in some sense. SMOTE is adding a spurious linear structure to the data, which I suspect is why they combined it with downsampling the majority class in the original paper (to limit the amount of synthetic points required, and therefore reduce the dominance of this spurious structure).

It is fairly easy to show that SMOTE generated points are not a good representation of the underlying data distribution. Train and test models with data where SMOTE has been applied (without changing class frequencies) to the training set after splitting into training and test sets, and do the same experiment but apply SMOTE before partitioning. It will usually give better results if you have split the data before partitioning, as that means the model can learn the spurious structure of the data that is due to SMOTE, and be rewarded for it because it has been added to the test data as well. If SMOTE accurately represented the data, it wouldn't make any difference.

I suspect that doing something like a Parzen density estimator and sampling from that is likely to be more effective in most problems.

$\endgroup$
1
  • 1
    $\begingroup$ (+1) The straight line issues is the most obvious one. To that extent, I would add that SMOTE tries to interpolate linearly in a space we have little reason to actually believe it is linear too.. $\endgroup$
    – usεr11852
    Nov 3, 2022 at 3:18
0
$\begingroup$

Here's how I think of it. We have data $(X,Y) \sim P$ and we want to minimize the risk $R[f] = \text E_P[L(f(X), Y)]$ over some function space $\mathscr F$. We do this by minimizing the empirical risk $$ \hat R_n[f] = \frac 1n \sum_{i=1}^n L(f(x_i), y_i). $$ Under certain assumptions the argmin of $\hat R_n$ over $\mathcal F$ converges to the argmin of $R$ so this is an appropriate model selection procedure.

If we have $y_i \in \{0,1\}$ for a classification problem then we can rewrite the empirical risk as $$ \hat R_n[f] = \frac 1n \left(\sum_{y_i = 0} L(f(x_i), y_i) + \sum_{y_i = 1} L(f(x_i), y_i)\right) \\ = \frac{n_0}{n_0 + n_1} \cdot \frac 1{n_0} \sum_{y_i = 0} L(f(x_i), y_i) + \frac{n_1}{n_0 + n_1} \cdot \frac 1{n_1} \sum_{y_i = 1} L(f(x_i), y_i) $$ so this is a convex combination of the empirical risks on each class.

I'll assume the positive class is the minority class. If we have too few examples with $y_i=1$, then that empirical risk will be a poor estimate. So one natural approach is to try to bolster it with synthetic samples.

This means that our goal is to produce new data points coming from the conditional distribution $X \mid Y = 1$. The better these points are, the better we'll do.

One strategy: estimate $X \mid Y = 1$ via the empirical distribution, i.e. bootstrap. This is what we are doing when we oversample the minority class.

Another option: get new $X$ by sampling along the lines connecting each pair of minority $x_i$. SMOTE does this. If $X \mid Y=1$ is well-approximated by this distribution, which will happen when we'd want to assign $+1$ to the convex hull of the $x_i : y_i = 1$, then SMOTE will probably work well. There's no free lunch and sometimes this will work and other times it won't.

$\endgroup$
8
  • $\begingroup$ So don’t bother trying to synthesize majority-class points? $\endgroup$
    – Dave
    Aug 11, 2022 at 15:41
  • $\begingroup$ @Dave oh we could do that too, I think there's just usually less interest in that. But the exact same points apply $\endgroup$
    – jld
    Aug 11, 2022 at 15:44
  • 2
    $\begingroup$ @Dave my main point is that it's all about how well the synthetic distribution approximates the conditional distribution of $X \mid Y=1$, and whether or not that's the case is a typical no-free-lunch thing like "does a linear model work well" where the best answer is just "sometimes". I also personally find a lot of insight from connecting SMOTE to oversampling as I do above, so we can see that they are different approaches to the same problem of picking a conditional distribution $\endgroup$
    – jld
    Aug 11, 2022 at 15:45
  • $\begingroup$ The justification for SMOTE was partially that classifiers (especially the rather primitive ones they used) overfit if we heavily oversample the minority class, so smearing the data out a bit effectively regularises (that's my view of it, rather than Chawla's) the solution. The problem is that this is only a problem in settings where data are scarce, which means that asymptotic results are not that reassuring. It would be much more sensible if SMOTE generated patterns at random withing the convex hull of a subset of points, but that isn't what it does. $\endgroup$ Aug 11, 2022 at 15:52
  • $\begingroup$ The problem with convex hull type justifications is that it introduces a bias as in practice new real samples will not all lie within the convex hull of your intial points, so that is likely to push the decision boundary towards the minority class, but in a very heuristic manner with little or no theoretical justification. $\endgroup$ Aug 11, 2022 at 15:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.