Cross-validation with unbalanced-classes

Question

I'm a little confused on how to manage my data set with WEKA.(for data mining)

I have a Dat set including 11377 record classified as follows:

• 11111 records have class YES
• 266 records have class NO

This is an unbalanced class, and if i start the classification process with WEKA, the results will be poor. I want to use the Cross-validation with 10 fold for the classification of data with J48 tree algorithm, but first i need to oversample my minority class? How i can prevent overfitting of data?

I would like to know how I should get this situation to get a good analysis. Thanks in advance!

Answer 1

Start by making sure that you are performing the oversampling in the correct portion of your analysis. First you create a validation set that is not at all involved in the training by randomly pulling out some percentage of your data. Make sure that randomly sampled data has the correct proportion of YES and NO. Then you use the remaining data [only] (do NOT use the data in the validation set) to perform your oversampling and analysis.

Your randomly selected validation set is then used to determine the performance of your model. Check the selectivity and specificity of your results. If you want to get a general idea of how much overfitting there is you can see how the model performs against the training set and then compare that to how it performs against the validation set. The validation set will give your anticipated real world performance.

Essentially...I'm saying that you can check for overfitting by keeping a validation set separate and comparing against your training set. In order to actually prevent the overfitting you'll have to review your data and do feature selection carefully.

Answer 2

To complete Adam Sampson answer, I should add that (not) oversampling the validation set is regard to the metric you use and the minority class.

If the POSITIVE in minority and you want to maximize the F1, recall, or precision, you do not need to oversample the validation set; but if NEGATIVE is minority (like you case with 266 NO), you may need to do Random/SMOTE oversampling. However, you can consider NO as POSITIVE and skip validation set oversampling if your metric is F1, recall, or whatever deals with TRUE POSITIVE.

Answer 3

For this dataset, I think you should first randomly select, for instance, 20% of your dataset as the test set. Then for the remaining 80% of your dataset, you do the oversampling(SMOTE, etc) first and apply 10 fold cross-validation Fine tune each classifier on this balanced dataset. And use the remaining 20% of your dataset to compare the results from all of your classifiers.

You could check this link for more information: Do we need a test set when using k-fold cross-validation?