How to deal with factors with rare levels in cross-validation? Suppose in a regression analysis in R, I have a factor type independent variable with 3 levels in my train dataset. But in the test data set that same factor variable has 5 levels. Therefore I can not predict the response values for test dataset. What should be done in this case?
 A: As a very first thought, this means that at least your training set is not representative for the application data. Whether the test set is representative is a question that you should IMHO think very carefully about. In this context, it is also important to find out whether these missing classes are a problem of the particular training set being too small, or whether this is a general characteristic of the problem/task/application. I.e., whether new classes that have never been encountered before will be appearing all the time.
In principle, I see two possibilities of dealing with this situation:


*

*Say that the training set is for sure not representative and ask for more data, particularly for data of the missing classes. This does make sense in case you come to the conclusion that the problem lies with the particular training set, not with the general characteristics of the application.

*In any case, knowing that the training data misses classes, I'd consider using a one-class classifer. I.e. a classifier that treats each class independently of any possible other class. Ideally, a one-class classifier should return "unknown class" for the test cases of classes that have not been available for training. For one-class classifiers, testing this "rejection" of cases belonging to truly unknown classes does actually make sense.

edit wrt @gung's comment: I assume that the train/test split is fixed for some hopefully good reason.
