# k fold cross validation: nominal predictor level appears in the test data but not the training data

I wrote my own cross validation function for model output in R (lm, glm and so on; named MOD_k_fold_r2 in my R package). It works for nominal predictors as well. In a few rare cases, a level of the nominal predictors does not appear in the training data, but does appear in the test data. When this happens, the predict function throws an error because it doesn't know which beta to assign to this new level and therefore it can't predict the Y values for the cases.

Is there any standard way to deal with this? Perhaps assign the beta a 0 coefficient manually? This would require scoring the new cases manually instead of using the predict function I think. Currently, I am instead just disregarding these runs that produce errors and rely on the remaining runs. This is a rare problem for most datasets.

Update

Proposals so far:

• Skip runs that use test data that have levels that do not appear in the training data.
• Use stratified sampling to make sure that all training datasets include at least one case with each level.
• If levels are missing, then insert pseudocases with arbitrary datapoints, e.g. mean of all values.
• One way to avoid this is to deliberately form the training set such that all levels of each nominal predictor are represented at least once in the training set. I have no idea what the "standard" way of dealing with this is though. – Jake Westfall May 22 '16 at 20:10
• @JakeWestfall: "stratification" should be a start point to look this up. – cbeleites May 31 '16 at 7:05