Suppose that I am doing a multiclass classfication problem where some of the classes only occupies a very low percent of the whole training data. It is possible that when I divide it into k folds, some of the folds will not cover all possible labels. What should one do in this case?
1 Answer
$\begingroup$
$\endgroup$
Yes, it is possible this could happen, especially if you have some classes that have very few representatives. One way to approach it is to do a stratified cross-validation, in which you maintain as closely as possible the relative proportions of each class. There are tools in R and Python that enable this without having to write a lot of extra code yourself. If you have a class that is particularly small, this might be difficult to do, however.
