0
$\begingroup$

I have an unbalanced dataset and would like to apply SMOTE to the training data. I can either do one of the following:

  1. Inside trainControl() add sampling = "smote" and then run train()
  2. First sample the training data using SMOTE(), NOT include sampling in trainControl() and then run train(). For SMOTE() I used the default parameters as in the documentation: SMOTE(form, data, perc.over = 200, k = 5, perc.under = 200, learner = NULL, ...)

However, I end up with different results because the training dataset are different sizes, the first option maintains the number of observations, but the second option reduces the number of observations.

I would like to know why and which one would be the correct way of doing it. Thanks.

$\endgroup$

closed as off-topic by kjetil b halvorsen, Michael Chernick, Jeremy Miles, mkt, Christoph Hanck Feb 1 at 9:40

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "This question appears to be off-topic because EITHER it is not about statistics, machine learning, data analysis, data mining, or data visualization, OR it focuses on programming, debugging, or performing routine operations within a statistical computing platform. If the latter, you could try the support links we maintain." – Michael Chernick, Jeremy Miles, Christoph Hanck
If this question can be reworded to fit the rules in the help center, please edit the question.

  • $\begingroup$ Are you sure you need balancing? Whats the ratio of your class sizes? What's the size of your data set? How are you evaluating your model performance? I'm sorry to redirect the question, but these balancing techniques are vastly overused, and often a different approach that does not mangle your training data is better. $\endgroup$ – Matthew Drury Jan 30 at 17:15
  • $\begingroup$ This is a fraud dataset, being 99% not fraudulent and 1% fraudulent, very unbalanced. The dataset is composed 4500 observations. I will build models using logistic regression, classification tree and random forest. To compare the models I will be using Kappa since it is more appropriate for unbalanced classes. $\endgroup$ – Fernando Han Jan 30 at 18:57
  • 1
    $\begingroup$ Since you're predicting fraud, you should almost certainly NOT be viewing it as a classification problem, instead you almost certainly want to think of this as a risk estimation (what is the probability this is fraudulent) or a ranking (what is the most likely to be fraudulent) problem. It's generally not correct in this circumstance to try to classify each observation as fraud or not fraud. Class balance is not a very relevant concern in risk estimation or ranking problems, so I'll stick to the point that SMOTE is not useful in this problem. $\endgroup$ – Matthew Drury Jan 30 at 20:51
  • 1
    $\begingroup$ I haven't found class balancing techniques to ever be useful, personally (and I've worked with some ostensibly unbalanced data sets, half a percent or so of the positive class). I've always found it better to fit the model optimizing for probabilistic measures like log-loss (for which class balancing is not really relevant), and then set a classification threshold when I need to construct a decision rule. There may be situations where one class is exceedingly rare that it is important, but that's outside my experience. $\endgroup$ – Matthew Drury Jan 30 at 21:42
  • 2
    $\begingroup$ Possible duplicate of SMOTE data balance - before or during Cross-Validation $\endgroup$ – G5W Jan 31 at 17:54
1
$\begingroup$

See the documentation.

You can optimistically bias the results when sampling outside of resampling. A synthetic version of a training set point can end up in the test set.

$\endgroup$
  • $\begingroup$ Great documentation, it explains exactly my question. Thank you so much! The reason I am asking this is because I want to run a Chi-square test (goodness of fit) on different logistic regression models of different complexity (less variables) but the anova() function does not work with caret models built with train. So I am having to sample outside and then run the glm() function. As a follow-up question, is there a way either to use smote in resampling using glm() or do chi-square test using caret model? $\endgroup$ – Fernando Han Jan 30 at 19:33

Not the answer you're looking for? Browse other questions tagged or ask your own question.