General question on oversampling

Question

I once read (unfortunately I cannot find the source anymore) that oversampling is only "allowed" when the class distribution is very, very skewed, i.e. 1:99 or worse.

So in my case I have a class distribution of 30:70. When don't apply oversampling in my neural network (10-20-1, sigmoid (hidden layer), softmax(output layer)) then the f-measure is around 0.10 and the kappa around 0.1, which means in fact nothing to predict/features very bad.

When I apply oversampling, so the class distribution is almost 50:50 the f-measure goes up to 0.9 and the kappa to 0.85. As this sounds too good to be true, I think oversampling in this case isn't valid.

What is your opinion this? And do you have a good reference for rules of thumb on oversampling?

Answer 1

To generalize your question, I believe you're saying you oversampled a high influence region of a predictor and you're now wondering why your model performance measures seem much, much better. If this were a linear regression model, it would make perfect sense since the measures of model performance there are generally improved when the variance of the sampling distribution of the input variable is increased. I think your neural network is no different.

Oversampling can be done for efficiency reasons, such as the above scenario. However I would be very leery of doing so based on a preliminary analysis of the data since the confirmatory model will be subject to overfitting or bias. If you're interested in the population, you don't want a select group to drive the trend. Consider SES and health disparities. If you oversample high earning families, you will also capture their considerably better health outcomes. That could "torque" the least squares regression slope about the central mean of the bivariate distribution of SES/health measures and make the fitted mean health outcomes for low income families seem much lower than their actual mean(this is a case of model misspecification).

There are methods of stratified analysis in which you might sample systematically according to a prespecified "importance sampling distribution" or weighting scheme. But to do inference on the population of interest, you need to consider how to reweight your analysis to account for this sampling strategy.