Does it make sense to use cross-validation with factor variables that have 3+ levels? When using bestglm, I get an error saying that it doesn't work with categorical variables. In the documentation the reasoning is "Cross-validation is not available when there are categorical variables since in this case it is likely that the training sample may not contain all levels and in this case we can’t predict the response in the validation sample." but I can't find anyone else discussing this issue.
That's too bad. Computer software is notoriously bad about intelligently handling individual variables or features that may be parsed out into two or more columns in a model matrix. Part of me wonders if it's intentional, as if to make the user ask themselves the hard questions. Categorical variables are certainly an example of that. Sophisticated nonlinear models for continuously variables, like splines, also suffer that issue.
The issue here is not with the particular objective function (e.g. CV reliability, or AIC, or BIC), but with the particular stepwise model selection strategy being employed to "step" through the family of parametric models. You can "drop" or "add" a single column anymore, you have to "drop" or "add" the 2+ columns corresponding to the 3+ level factor simultaneously.
It's something that can easily be done with "by-hand" optimization, depending on your R coding skills. I don't know what goes on under the hood with
bestglm, but you'll run into the same issues with forward and or backward model selection.
A few practical workarounds for R novices would be the following:
1) create indicator variables for each factor level. You might drop 1 level of the factor in the bestglm, but not the other... leads to very confusing results.
2) factors that are pseudocontinuous or ordinal can just be coded as continuous variables.
3) [best option] instead of using an automated model selection process, propose a few practical models, and potentially use CV or BIC or ROCs to compare the relative utility of each model.