2
$\begingroup$

I have a highly imbalanced (~0.4% minority class) binary classification dataset of time series (flux) observations, and am now at a loss on how to classify it, as the data aren't separable. Here's what the data look like, both originally and with SMOTE oversampling.

enter image description hereenter image description here

I have tried SVM, KNN, RF, ANN, and CNN. SVM, KNN, ANN, and CNN produce similar results—classifying both training classes correctly about 60% of the time, but only classifying the test minority class correctly 1-2% of the time. RF overfits with 100% training accuracy/recall/precision, but the same results on the test data. I get these results regardless of the resampling technique I use on the training data.

Is it possible to do better than this, i.e., classify the minority class more than 2% of the time on unseen data? The classes pretty much overlap, so is there a way to separate them? I've seen some threads mentioning probability calibration, which I am familiar with. Is this something worth trying out even though I am only predicting class labels?

$\endgroup$
1
  • 1
    $\begingroup$ 0. Welcome to CV.SE. 1. You show the PCA projections, maybe there are other projections that suggest a better separability, (e.g.t-SNE or UMAP) but importantly your minority samples appear to significantly overlap so it is very hard to recognise any decision boundary. 2. Doing EDA and using your own subject matter expertise, can you distinguish between the two classes? Maybe there is a feature engineering step you could do that would really help the classification task. $\endgroup$
    – usεr11852
    Jul 3, 2022 at 15:38

1 Answer 1

2
$\begingroup$

I suspect that there is little you can do here as there isn't enough data of the minority class to properly characterise the underlying distribution. One of the problems with SMOTE is that the way the synthetic data are constructed generates spurious structure in the data (for instance the hole centred around comp-1 = 0 and comp-2 = -2). For this reason, make sure you perform evaluation using the original data distribution, not the SMOTEd data as the latter will reward learning of this spurious structure. Another spurious structure that SMOTE creates is that all synthetic patterns will lie within the convex hull of the original data, which creates the straight, very defined boundaries, when in practice, there would be some chance of minority patterns beyond that boundary if sampled from the true data generating process. SMOTE is a sensible sounding heuristic, but it is far from ideal.

Class imbalance generally isn't a problem unless you have very little data, and SMOTE is unlikely to improve performance in most cases. Unfortunately where there is very little data, learning algorithms can demonstrate an undue bias against the minority class and in principle SMOTE can help, but if you have that little data there will be no way to determine the optimal amount of resampling to apply to correct that bias.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.