I am working with a fairly unbalanced dataset (event class < 5% - it's a binary classification problem). To deal with this imbalance, I am trying out various techniques such as Oversampling the minority class (as well as synthetically generating samples using SMOTE), Under-sampling the minority class etc.
The problem that I am facing is, there is really no linkage between the performance measure than I get on my training sample (synthetically balanced) vis-a-vis what I get on my test sample. I am aware that oversampling can lead to inflated measures of performance, but the difference is stark (I am looking at Kappa statistic - On my training sample, I am getting a value in the range of 0.6-0.7, whereas on the test set it drops to less than 0.1). So my questions are:
a) Is there something than I can do to reduce this deviance between train and test perf. measures? I have been trying out different "ratios" (event/non-event ratio) while sampling, but this hasn't really helped.
b) Given this huge difference, is oversampling even a valid technique to pursue, in this particular case?
c) Finally, I had also tried the other recommended approach for dealing with unbalanced classes (by providing class weights, where the algorithm permits this functionality). This reduces the difference between train/test sets, but then the performance measures are not good for both of these sets! And hence, I was trying out the sampling techniques.