I once read (unfortunately I cannot find the source anymore) that oversampling is only "allowed" when the class distribution is very, very skewed, i.e. 1:99 or worse.
So in my case I have a class distribution of 30:70. When don't apply oversampling in my neural network (10-20-1, sigmoid (hidden layer), softmax(output layer)) then the f-measure is around 0.10 and the kappa around 0.1, which means in fact nothing to predict/features very bad.
When I apply oversampling, so the class distribution is almost 50:50 the f-measure goes up to 0.9 and the kappa to 0.85. As this sounds too good to be true, I think oversampling in this case isn't valid.
What is your opinion this? And do you have a good reference for rules of thumb on oversampling?