oversampling in nested cross validation

Question

Introduction

I have a small mixed dataset consisting of continuous and categorical independent variables with a dichotomous dependant variable. I'm running various algorithms (neural networks, random forests and logistic regression) to find the best model to generalise on unseen data. To do this i have performed a nested cross validation so that I can train and optimise hyper parameter and feature selection (on the inner loop) without being too biased on the model performance. I have conducted a 10 fold cross validation in the inner loop and outer loop.

My Question

My dataset has a class imbalance. To overcome this problem I have oversampled with replacement the minority class so that the majority is now at 80% and the minority at 20%. I know this still has imbalance, however it is now more appropriate to what can be found in the field and better than the original imbalance. My question is will this cause bias in the model performance because I used a oversampling technique with replacement?

Answer 1

As far as I know, in spite of how widespread this idea is, there is no convincing argument that artificial oversampling makes sense or leads to a better model. See for instance this question, or this question, both about whether class imbalance is a real problem, both highly voted/often viewed, neither with any clear indication that this practice is a good idea.

Oversampling will clearly introduce bias in your model because $P(class|data) = \frac{P(data|class)P(class)}{P(data)},$ and you're artificially messing with $P(class)$.