# SMOTE vs Stratified Sampling in highly imbalanced dataset - classification

I am working on a project with the goal of predicting Cerebral strokes from brain arteries data (speed of blood, resistance etc. of one artery and of the neighboring ones).

I have a dataset with labeled data but it's highly imbalanced: patients with stroke represent a minority, hence the models (tried RF, & some boosting) predicting always 'non stroke'.

I am looking for the most efficient ways for dealing with that, and these are my options:

1. Tackle class imbalance - SMOTE or Stratified sampling? what are the differences between these two?

2. [BONUS] Tackle through a loss function penalizing false negative over FP - any idea of an efficient way to do that?

• You don't want to classify, you want to predict risk. Resampling the minority is going to ruin your risk estimates because the prevalence is different than what is observed in the population. See my answer here (stats.stackexchange.com/questions/558942/…) for more. Mar 31 at 22:41
• – Dave
Apr 1 at 1:22

## 1 Answer

Use a probabilistic classifier (e.g. logistic regression, kernel logistic regression or Gaussian process classifiers) that outputs the probability of a stroke rather than a hard 0/1 classification. That way you can factor in unequal misclassification costs without refitting the model (it is just a change of threshold).

In general, classifiers don't have an undue bias against the minority class, and if all patterns are being assigned to the majority class, that is the optimal decision for equal false-positive and false-negative costs (see my related question), so you don't need to do anything about the imbalance. The reason for balancing etc. is actually because the misclassification costs are not equal, but this has nothing to do with the imbalance per se, it is just that imbalanced learning problems tend to be ones with obviously unequal costs. For example, I suspect it is a far worse error to classify a patient as not being at risk of stroke when they are (they may go home untreated, have a stroke and die) than to classify them as at risk of stroke when they are not (they will just be subjected to unnecessary further testing). We should be considering misclassification costs anyway, regardless of imbalance.

If you have a very small dataset, then the classifier may have an undue bias against the minority class (see my answer to this related question), in which case it principle it would be worth resampling or re-weighting to compensate for this bias, however as the data are very scarce it is difficult to judge how much compensation is required, so collecting more data to get rid of the bias is likely to be the only reliable solution.

I disagree with some others on this issue, but it is possible for discrete classifiers to give better 0/1 classifiers than probabilistic methods. This is because they focus on the contour of the probability of class membership that actually matters and are less likely to be distracted by modelling features of the probability of class membership that don't affect the 0/1 classification. For a synthetic example, see my question here. In that case, you may want to try discrete classifiers, and some of those cannot accommodate unequal misclassification costs, and so resampling the data may be required. Note however, I asked a question (with a small bonus) asking for practical examples where this worked and it went unanswered.

Lastly I would advise against using SMOTE for modern classifiers that can accommodate unequal misclassification costs (like the SVM where you can have different $$C$$ values for the positive and negative classes) and which have a means of dealing with overfitting (again for the SVM the $$C$$ parameters and also the kernel parameters). The SVM has a lot of theory behind it, but the regularisation implemented by SMOTE by blurring the training examples is only heuristic and is only really likely to be beneficial if you are using a very basic classifier, like a decision tree.

• Minor note: These is not a probabilistic classifier; it is a probability estimator. And I've never seen anyone who studied statistics for a few years want to use SMOTE. Apr 2 at 14:41