Consider a binary classification problem. As far as I know, if the dataset is imbalanced and if the two classification errors are not equally serious, then we should balance the distribution of the classes in the training set so that 50% of the observations in the training set belong to class A and the remaining 50% belong to class B. The test set, used to estimate the performance of the fitted model on the training set, should not be balanced instead. Some classification models have one or more tuning parameters which must be chosen carefully. One way to do this is by performing cross validation on the training set. So, in the first step of a k-fold cross validation for parameter tuning, we take the training set and split it in k parts. In the first step of cross validation, the first part of the data will be used to estimate the performance of the models fitting on the remaining k-1 parts with different values for the tuning parameters. Of course, this k-1 parts should be balanced. But should the first part of the data be balanced too? Thank you.
1 Answer
I don't see how an artificially balanced set can be representative for the actual population of the application.
There are reasons to start working with balanced data, e.g.
- if you have to start with a very limited sample size , it may be good to have case-control matching or at least equal sample sizes (see below)
- the relative frequencies in the relevant ground population may be actually unknown - so equal relative frequencies is your best bet;
- or you have reasons to suspect that the sampling process produces a bias which is the cause of the imbalance.
However, at some point, you'll have to find a model that can actually deal with the underlying imbalance - otherwise, you'll never get an applicable solution for the problem at hand. Go for that immediately, and all problems with the cross validation are solved as well.
As for the validation results: you can always measure the prediction quality for each class separately and then weight these results according to the seriousness of the different types of misclassification (cost function) and according to the relative frequencies of the classes (e.g. as you need to do when converting sensitivity and specificity into positive and negative predictive values). Error propagation should give you a practical guesstimate of the relative test sample sizes you need for the different classes.
But for very small sample sizes, the variance on the test results due to limited numbers of test cases may be the limiting factor. In that case, you need to make sure that no single class has overwhelming dominance on the validation uncertainty because of too few cases. On the other hand, if your sample sizes are so limited, you should maybe avoid data-driven model tuning at all.
if the dataset is imbalanced and if the two classification errors are not equally serious, then we should balance the distribution of the classes in the training set so that
This can obviously help only if the cost function says the serious misclassification is not recognizing the small class. Even then, I'd prefer to train on more (all) data, and use a classifier where I can include the cost function.
Making the data fit the needs of the pet classifier (instead of choosing the model according to the characteristics of the application) sounds awfully like a Procrustes approach...
