I wrote my own cross validation function for model output in R (
glm and so on; named
MOD_k_fold_r2 in my R package). It works for nominal predictors as well. In a few rare cases, a level of the nominal predictors does not appear in the training data, but does appear in the test data. When this happens, the
predict function throws an error because it doesn't know which beta to assign to this new level and therefore it can't predict the Y values for the cases.
Is there any standard way to deal with this? Perhaps assign the beta a 0 coefficient manually? This would require scoring the new cases manually instead of using the
predict function I think. Currently, I am instead just disregarding these runs that produce errors and rely on the remaining runs. This is a rare problem for most datasets.
Proposals so far:
- Skip runs that use test data that have levels that do not appear in the training data.
- Use stratified sampling to make sure that all training datasets include at least one case with each level.
- If levels are missing, then insert pseudocases with arbitrary datapoints, e.g. mean of all values.