Sampling method for imbalanced data?

Question

I have an imbalanced dataset with 4995:5 ratio as well as other datasets with less imbalanced ratios. I split this 4995:5 ratio into training and testing for about 2/3 training and 1/3 testing. I also decided to downSample using caret for the 4995:5 ratio dataset - this dataset now becomes 5:5.

Repeated cross validation works fine for the other datasets since there are more of the minority class, but for the training set of the 4995:5 ratio, I get the binomial class has less than 8 observations for either random forest or logistic regression.

Would I have to resort to bootstrap or LOOCV? This dataset seems to be problematic because of the terrible ratio.

Answer 1

If the number of samples in the class is just 5, it seems that machine learning is not a good approach to the problem. Even if we ignore the problems in training the classifier (and you'll need on of very low VC dimension), validating it on 5 samples will not give you enough confidence in the results. I think that relaying on domain knowledge to build a (small and simple) classifier while using the dataset for validation will work better in this setting.

I didn't understand the relations between this dataset and the other data sets you mentioned. If you have a more reasonable number of samples you can try the method described here: https://www.quora.com/In-classification-how-do-you-handle-an-unbalanced-training-set/answer/Dan-Levin-2

Answer 2

I assume that 4995:5 is absolute number of samples, not just ratio. You don't have enough data to do a holdout, you should do some form of CV that will keep most of your samples from the smaller group in your training data. So LOOCV or stratified 5 fold CV. That being said, with 5 samples, you have probably bigger problems than validation.