Suppose in a regression analysis in R, I have a factor type independent variable with 3 levels in my train dataset. But in the test data set that same factor variable has 5 levels. Therefore I can not predict the response values for test dataset. What should be done in this case?

  • 1
    $\begingroup$ This isn't primarily about how to use R. & if it were only about how to use R, it would be off-topic here. There is a good statistical question here, though. $\endgroup$ – gung - Reinstate Monica Mar 27 '15 at 16:24
  • 1
    $\begingroup$ Can you please add some sample data? It will be easier to address this way. $\endgroup$ – Andrew Owens Mar 27 '15 at 16:25

As a very first thought, this means that at least your training set is not representative for the application data. Whether the test set is representative is a question that you should IMHO think very carefully about. In this context, it is also important to find out whether these missing classes are a problem of the particular training set being too small, or whether this is a general characteristic of the problem/task/application. I.e., whether new classes that have never been encountered before will be appearing all the time.

In principle, I see two possibilities of dealing with this situation:

  • Say that the training set is for sure not representative and ask for more data, particularly for data of the missing classes. This does make sense in case you come to the conclusion that the problem lies with the particular training set, not with the general characteristics of the application.

  • In any case, knowing that the training data misses classes, I'd consider using a one-class classifer. I.e. a classifier that treats each class independently of any possible other class. Ideally, a one-class classifier should return "unknown class" for the test cases of classes that have not been available for training. For one-class classifiers, testing this "rejection" of cases belonging to truly unknown classes does actually make sense.

edit wrt @gung's comment: I assume that the train/test split is fixed for some hopefully good reason.

  • $\begingroup$ Would it make sense to partition your data w/i each level & then combine those into the folds? Eg, X1 has 2 levels w/ 90 & 10 data; you could partition the 10 into 10 singletons & the 90 into 10 sets of 9, then you'd combine one set from each to form each of your 10 folds for CV. This is what came to mind when I read the Q, but it doesn't seem you're suggesting that. Is this a valid solution? (I could ask as a new Q, if you prefer.) $\endgroup$ – gung - Reinstate Monica Mar 29 '15 at 17:55
  • $\begingroup$ @gung: I think logically it is a separate question, yes. Also I see a huge principal difference between training set not covering all classes and ensuring all classes of a given small data set show up in training and test splits (= stratification). $\endgroup$ – cbeleites supports Monica Mar 29 '15 at 18:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.