1
$\begingroup$

In class we are learning about the SMOTE (Synthetic Minority Oversampling Technique) algorithm. As I understand, this algorithm can be used to increase the effectiveness of Machine Learning models when the datasets are imbalanced.

An example our professor gave was that suppose you have a dataset with information belonging to medical patients (e.g. age, gender, height, etc.). Suppose that the response variable is whether or not the patient has a specific disease or not - thus, the goal would be to predict whether or not a new patient has a disease (i.e. supervised binary classification). Now, imagine that this is a rare disease and only 5% of the patients in your dataset has the disease. If you fit a Machine Learning model to this data (e.g. Random Forest), the model can "get away" with saying that all patients do not have the disease - and still produce a very good accuracy (although the F-score will be bad)! This is because the Machine Learning model will not have observed enough disease cases to effectively "learn" the difference between diseased cases and non-diseased cases - and therefore poorly generalize, and state that all future patients do not have the disease.

In practice, the SMOTE algorithm is allegedly able to partly rectify this problem. Suppose there is a dataset of 100000 patients and 5% of the patients (5000 patients) have the disease. Suppose you randomly select 10000 patients (10% of the original dataset) who have the disease and the 5000 patients that have the disease - now, if you create a new dataset with these 15000 rows, you might be able to create enough of a "contrast" in such a way that the Machine Learning model will have a better way of distinguishing between Disease and Non-Disease cases. What is described here is a general "Oversampling/Undersampling" technique - the SMOTE algorithm is a more sophisticated version of this same premise.

I have the following question: When reading online, the SMOTE algorithm is often used in more Machine Learning Model contexts compared to "Traditional Statistical Models" contexts. For example, I see SMOTE being used more alongside models such as XGBOOST and Random Forests when compared to SMOTE being used for Regression Models. Suppose in this above example with the medical patients and the imbalanced data, I want to use a Logistic Regression model for interpretability reasons (e.g. Odd's Ratio, Statistical Significance of Coefficients, etc.) : In theory, is there anything which suggests that Oversampling/Undersampling approaches are inherently ill-suited for a Logistic Regression model? Or should such approaches only be used alongside Machine Learning models?

$\endgroup$
5
  • 1
    $\begingroup$ That you’ve added a bounty tells me that you are not satisfied with my post. As I thought I was quite thorough last month, I am wondering what more you would want. $\endgroup$
    – Dave
    Commented Dec 29, 2022 at 13:30
  • $\begingroup$ @ Dave: would just be curious to hear other opinions about this topic as well $\endgroup$
    – stats_noob
    Commented Dec 29, 2022 at 13:31
  • $\begingroup$ @stats_noob The bounty message says "stats_noob is looking for an answer from a reputable source" so it sounds like you're asking for an answer that cites a peer-reviewed source or other publication. Is that correct? Because your comment, which says "curious to hear other opinions about this topic as well" sounds like a it asks for a different kind of answer than "answers from a reputable source." How should potential answer-writers reconcile these facts? $\endgroup$
    – Sycorax
    Commented Jan 3, 2023 at 1:55
  • $\begingroup$ I asked a question with a modest bonus if anyone could provide an example where resampling the data results in a demonstrable improvement in accuracy ( stats.stackexchange.com/questions/559294/… ). So far, there have been no answers, which I think tells it's own story. There are good reasons for resampling data, but class imbalance (per se) is not generally one of them. $\endgroup$ Commented Apr 5, 2023 at 14:28
  • $\begingroup$ In the case of logistic regression, resampling the data according to the labels asymptotically only changes the bias parameter of the model. So SMOTE is even less pointful for linear logistic regression. $\endgroup$ Commented Apr 5, 2023 at 14:30

1 Answer 1

7
+50
$\begingroup$

Class imbalance isn’t much of a problem, and many of the apparent issues come from using a surprisingly poor “accuracy” metric. Depending on how much you care about this class and topic, you might want to press your instructor on why class imbalance is an issue at all.

Moreover, SMOTE does not seem to be very good at what it aims to do, even.

So should you use SMOTE when you run a logistic regression? Probably not, but that’s really because you probably shouldn’t use it for any kind of machine learning. I would not consider logistic regression special, however, in using SMOTE, and I would expect the full-credit answer on your exam to be that SMOTE is totally compatible with logistic regression (even if I only consider this the full-credit answer and not the correct answer).

You also mention undersampling. Since class imbalance isn’t really a problem, there is no need to discard precious data to fix a non-problem.

(Yes, Dikran Marsupial gives an interesting example of when class imbalance really is a problem. That’s quite different from what most in machine learning mean when they talk about class imbalance being a problem. I might even argue that example to be a matter of experimental design rather than of model evaluation, the latter of which is where practitioners like your instructor seem to be under the impression that class imbalance poses problems.)

EDIT

The bounty message mentions wanting a reputable source. Frank Harrell’s Regression Modeling Strategies textbook gets into this and has many references to the primary literature. Further, he has (at least) two blog posts on the topic: (1) (2).

EDIT 2

I contest the example your professor gave about a naïve classifier with an accuracy of $95\%$ is giving a good accuracy score. Where I think people get in trouble with this is interpreting accuracy as being akin to $R^2$ in regression, since both are proportions (and that isn’t even totally true for $R^2$). Their logic seems to be that, since $R^2 = 0.95$ probably would be considered strong performance, accuracy of $95\%=0.95$ should be considered impressive, too. The trouble is that all $R^2$ and accuracy have in common is that they are (sort of) proportions. A reasonable view of $R^2$ is that it is a comparison of the square loss of the model and the square loss of a “must-beat” naïve model. A reasonable counterpart for $R^2$ in terms of classification accuracy could be a comparison of model error rate to the error rate of a model that naively guesses the majority class every time. Subtracting from one the ratio of these two error rates then would be analogous to $R^2$, and the performance of the model given by your professor would be zero, correctly highlighting the fact that the model basically does nothing, despite an accuracy score of $95\%$ that (misleadingly) looks like a high $\text{A}$ in school that makes us happy. Why such a metric is not more widespread is a mystery to me. Indeed, even a good UCLA page gives this metric (granted, in a different (but equivalent) form as something like $R^2$ for a classification problem, yet I rarely see it discussed.

$\endgroup$
5
  • $\begingroup$ @ Dave: Thanks for your answer! This previous post here suggests that "oversampling based approaches" might work for such a problem? stats.stackexchange.com/questions/6067/… $\endgroup$
    – stats_noob
    Commented Nov 24, 2022 at 3:55
  • 3
    $\begingroup$ @stats_noob There are subtleties in that post and the one I linked by Dikran Marsupial. Yes, there are times when class imbalance can present challenges, but they are very much not the mainstream of machine learning or when most practitioners see class balance and jump to artificial balancing. That is, I still challenge your professor’s assertion that class imbalance is an inherent problem for a method like SMOTE to fix (and, as mentioned in my answer, it does not seem that SMOTE is even very good at synthesizing data). $\endgroup$
    – Dave
    Commented Nov 24, 2022 at 7:31
  • $\begingroup$ Thank you for your reply! I guess your comments about SMOTE also apply to over/under sampling methods in general? $\endgroup$
    – stats_noob
    Commented Nov 24, 2022 at 8:17
  • $\begingroup$ @stats_noob I would mostly oppose oversampling, yes, especially if you’ve already collected the data. Undersampling makes no sense to me. Data are valuable; why discard data? $\endgroup$
    – Dave
    Commented Jan 4, 2023 at 19:41
  • $\begingroup$ +1, but I disagree that SMOTE shouldn't be used with any machine learning methods. The original paper was written in the context of very simple classifier systems (e.g. decision trees) that didn't have any reliable means of avoiding overfitting or implementing cost-sensitive learning. It does serve a useful purpose for those sorts of classifiers. I wouldn't use it with modern classifier systems, e.g. SVM (or ironically logistic regression ;o) which have regularisation and can implement cost-sensitive learning easily. $\endgroup$ Commented Apr 5, 2023 at 14:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.