I have a small mixed dataset consisting of continuous and categorical independent variables with a dichotomous dependant variable. I'm running various algorithms (neural networks, random forests and logistic regression) to find the best model to generalise on unseen data. To do this i have performed a nested cross validation so that I can train and optimise hyper parameter and feature selection (on the inner loop) without being too biased on the model performance. I have conducted a 10 fold cross validation in the inner loop and outer loop.

My Question

My dataset has a class imbalance. To overcome this problem I have oversampled with replacement the minority class so that the majority is now at 80% and the minority at 20%. I know this still has imbalance, however it is now more appropriate to what can be found in the field and better than the original imbalance. My question is will this cause bias in the model performance because I used a oversampling technique with replacement?

  • $\begingroup$ Why do you think the class imbalance is a problem, and what problem is the oversampling addressing? $\endgroup$ Oct 18 '17 at 16:57
  • $\begingroup$ Hi Matthew, Class imbalance is a problem because the algorithms im working with tend to favour the majority class and disregard the minority. Oversampling will hopefully prevent this bias in the model $\endgroup$
    – Martin
    Oct 18 '17 at 17:24
  • $\begingroup$ Can you say more about the algorithms you are working with? $\endgroup$ Oct 18 '17 at 18:33
  • $\begingroup$ Hi matthew, im working with the algorithms i mentioned in the original question $\endgroup$
    – Martin
    Oct 23 '17 at 21:26

As far as I know, in spite of how widespread this idea is, there is no convincing argument that artificial oversampling makes sense or leads to a better model. See for instance this question, or this question, both about whether class imbalance is a real problem, both highly voted/often viewed, neither with any clear indication that this practice is a good idea.

Oversampling will clearly introduce bias in your model because $P(class|data) = \frac{P(data|class)P(class)}{P(data)},$ and you're artificially messing with $P(class)$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.