I have an imbalanced dataset with 4995:5 ratio as well as other datasets with less imbalanced ratios. I split this 4995:5 ratio into training and testing for about 2/3 training and 1/3 testing. I also decided to downSample using caret for the 4995:5 ratio dataset - this dataset now becomes 5:5.
Repeated cross validation works fine for the other datasets since there are more of the minority class, but for the training set of the 4995:5 ratio, I get the binomial class has less than 8 observations for either random forest or logistic regression.
Would I have to resort to bootstrap or LOOCV? This dataset seems to be problematic because of the terrible ratio.