2
$\begingroup$

I have an imbalanced dataset with 4995:5 ratio as well as other datasets with less imbalanced ratios. I split this 4995:5 ratio into training and testing for about 2/3 training and 1/3 testing. I also decided to downSample using caret for the 4995:5 ratio dataset - this dataset now becomes 5:5.

Repeated cross validation works fine for the other datasets since there are more of the minority class, but for the training set of the 4995:5 ratio, I get the binomial class has less than 8 observations for either random forest or logistic regression.

Would I have to resort to bootstrap or LOOCV? This dataset seems to be problematic because of the terrible ratio.

$\endgroup$
2
$\begingroup$

If the number of samples in the class is just 5, it seems that machine learning is not a good approach to the problem. Even if we ignore the problems in training the classifier (and you'll need on of very low VC dimension), validating it on 5 samples will not give you enough confidence in the results. I think that relaying on domain knowledge to build a (small and simple) classifier while using the dataset for validation will work better in this setting.

I didn't understand the relations between this dataset and the other data sets you mentioned. If you have a more reasonable number of samples you can try the method described here: https://www.quora.com/In-classification-how-do-you-handle-an-unbalanced-training-set/answer/Dan-Levin-2

$\endgroup$
1
$\begingroup$

I assume that 4995:5 is absolute number of samples, not just ratio. You don't have enough data to do a holdout, you should do some form of CV that will keep most of your samples from the smaller group in your training data. So LOOCV or stratified 5 fold CV. That being said, with 5 samples, you have probably bigger problems than validation.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.