Is SMOTE any good at creating new points?

Question

Cross Validated has a pretty thorough debunking of class imbalance being an inherent problem for SMOTE to solve.

However, SMOTE is explicitly a method for synthesizing new points.

Is SMOTE any good at synthesizing new points? If so, is there any way to use SMOTE-style point synthesis to improve model performance, even if we respect the natural class ratio by synthesizing points from all classes, not just the minority class?

(I wrote “SMOTE-style point synthesis” because the “M” means “minority”, so it’s hard to argue that we’re doing SMOTE if we synthesize new points from the majority class, even if we use the same approach.)

Answer 1

No, probably not.

(i) SMOTE can only generate synthetic data points within the convex hull of a subset of points (and therefore will be within the convex hull of the full set of points). In the true distribution, it is likely that there will be points that fall outside this distribution. For example, if you split the data to form test and training sets, some of the test points are likely to lie outside the convex hull of the training set (especially in high dimensions). SMOTE will be fundamentally unable to synthesise points in those positions, and hence cannot be representative of the underlying distribution.

(ii) SMOTE generates points lying on straight lines connecting pairs of existing data points. It is extremely unlikely that the true distribution is structured in that way - for a start it means that the data points you have been given are "special" in some sense. SMOTE is adding a spurious linear structure to the data, which I suspect is why they combined it with downsampling the majority class in the original paper (to limit the amount of synthetic points required, and therefore reduce the dominance of this spurious structure).

It is fairly easy to show that SMOTE generated points are not a good representation of the underlying data distribution. Train and test models with data where SMOTE has been applied (without changing class frequencies) to the training set after splitting into training and test sets, and do the same experiment but apply SMOTE before partitioning. It will usually give better results if you have split the data before partitioning, as that means the model can learn the spurious structure of the data that is due to SMOTE, and be rewarded for it because it has been added to the test data as well. If SMOTE accurately represented the data, it wouldn't make any difference.

I suspect that doing something like a Parzen density estimator and sampling from that is likely to be more effective in most problems.

Answer 2

Here's how I think of it. We have data $(X,Y) \sim P$ and we want to minimize the risk $R[f] = \text E_P[L(f(X), Y)]$ over some function space $\mathscr F$. We do this by minimizing the empirical risk $$ \hat R_n[f] = \frac 1n \sum_{i=1}^n L(f(x_i), y_i). $$ Under certain assumptions the argmin of $\hat R_n$ over $\mathcal F$ converges to the argmin of $R$ so this is an appropriate model selection procedure.

If we have $y_i \in \{0,1\}$ for a classification problem then we can rewrite the empirical risk as $$ \hat R_n[f] = \frac 1n \left(\sum_{y_i = 0} L(f(x_i), y_i) + \sum_{y_i = 1} L(f(x_i), y_i)\right) \\ = \frac{n_0}{n_0 + n_1} \cdot \frac 1{n_0} \sum_{y_i = 0} L(f(x_i), y_i) + \frac{n_1}{n_0 + n_1} \cdot \frac 1{n_1} \sum_{y_i = 1} L(f(x_i), y_i) $$ so this is a convex combination of the empirical risks on each class.

I'll assume the positive class is the minority class. If we have too few examples with $y_i=1$, then that empirical risk will be a poor estimate. So one natural approach is to try to bolster it with synthetic samples.

This means that our goal is to produce new data points coming from the conditional distribution $X \mid Y = 1$. The better these points are, the better we'll do.

One strategy: estimate $X \mid Y = 1$ via the empirical distribution, i.e. bootstrap. This is what we are doing when we oversample the minority class.

Another option: get new $X$ by sampling along the lines connecting each pair of minority $x_i$. SMOTE does this. If $X \mid Y=1$ is well-approximated by this distribution, which will happen when we'd want to assign $+1$ to the convex hull of the $x_i : y_i = 1$, then SMOTE will probably work well. There's no free lunch and sometimes this will work and other times it won't.