Modelling with Unbalanced dataset I am working with a fairly unbalanced dataset (event class < 5% - it's a binary classification problem). To deal with this imbalance, I am trying out various techniques such as Oversampling the minority class (as well as synthetically generating samples using SMOTE), Under-sampling the minority class etc.
The problem that I am facing is, there is really no linkage between the performance measure than I get on my training sample (synthetically balanced) vis-a-vis what I get on my test sample. I am aware that oversampling can lead to inflated measures of performance, but the difference is stark (I am looking at Kappa statistic - On my training sample, I am getting a value in the range of 0.6-0.7, whereas on the test set it drops to less than 0.1). So my questions are:
a) Is there something than I can do to reduce this deviance between train and test perf. measures? I have been trying out different "ratios" (event/non-event ratio) while sampling, but this hasn't really helped.
b) Given this huge difference, is oversampling even a valid technique to pursue, in this particular case?
c) Finally, I had also tried the other recommended approach for dealing with unbalanced classes (by providing class weights, where the algorithm permits this functionality). This reduces the difference between train/test sets, but then the performance measures are not good for both of these sets! And hence, I was trying out the sampling techniques.
 A: In general, artificially biasing the sampling of parts of the train set (either directly, by weighting, or by synthesis), is problematic. 
Virtually every classifier also learns the a-priori label distribution. In Naive Bayes this can be seen very directly. In other techniques (e.g., logistic regression, classification trees, etc.), this appears too indirectly.

The cases where oversampling/overweighting makes sense, are different:
The Wikipedia entry, for example, states

The usual reason for oversampling is to correct for a bias in the original dataset. One scenario where it is useful is when training a classifier using labelled training data from a biased source, since labelled training data is valuable but often comes from un-representative sources.

and sites as an example a sample where ~67% are male, whereas they are known to compose ~50% of the population.
A different case where overweighting (possibly by oversampling) is when the missclassification penalties are unequal.

Two things to note when dealing with highly imbalanced data:


*

*When tuning parameters (using the train set), it might be important to use stratified cross validation. 

*It pays to ensure that the train set and test set are prepared by a stratified split as well. For highly-imbalanced data, you might also need to make the test set larger than you usually do, in order to reduce test noise.
