Cross-validation with dummy variables?

Does it make sense to use cross-validation with factor variables that have 3+ levels? When using bestglm, I get an error saying that it doesn't work with categorical variables. In the documentation the reasoning is "Cross-validation is not available when there are categorical variables since in this case it is likely that the training sample may not contain all levels and in this case we can’t predict the response in the validation sample." but I can't find anyone else discussing this issue.

That's too bad. Computer software is notoriously bad about intelligently handling individual variables or features that may be parsed out into two or more columns in a model matrix. Part of me wonders if it's intentional, as if to make the user ask themselves the hard questions. Categorical variables are certainly an example of that. Sophisticated nonlinear models for continuously variables, like splines, also suffer that issue.

The issue here is not with the particular objective function (e.g. CV reliability, or AIC, or BIC), but with the particular stepwise model selection strategy being employed to "step" through the family of parametric models. You can "drop" or "add" a single column anymore, you have to "drop" or "add" the 2+ columns corresponding to the 3+ level factor simultaneously.

It's something that can easily be done with "by-hand" optimization, depending on your R coding skills. I don't know what goes on under the hood with bestglm, but you'll run into the same issues with forward and or backward model selection.

A few practical workarounds for R novices would be the following:

1) create indicator variables for each factor level. You might drop 1 level of the factor in the bestglm, but not the other... leads to very confusing results.

2) factors that are pseudocontinuous or ordinal can just be coded as continuous variables.

3) [best option] instead of using an automated model selection process, propose a few practical models, and potentially use CV or BIC or ROCs to compare the relative utility of each model.

• Thanks. I'm new to automated model selection. Could anyone point me to any comprehensive discussions of why manual selection is superior? – user7340 Aug 11 '14 at 14:13
• – AdamO Aug 11 '14 at 18:57
• That appears to be about stepwise vs. manual selection. I'm interested in automated selection via best subsets or elastic net regression. – user7340 Aug 27 '14 at 16:28
• The problem is general, in particular for the issue of "features" that are actually a combination of features but that can be expressed as "parsimoniously fewer" features. Nets and lasso are not immune to this. Model selection sucks for inference and it's not that good for prediction unless you absolutely need to restrict to a specific number of features for a prediction model (rarely the case). I don't have a lot of good things to say about feature selection except in a few very contrived circumstances. – AdamO Aug 27 '14 at 17:28