This question is related to previous ones but I believe distinct. I am primarily interested in prediction and I have access to LASSO variable selection (but without factor level grouping) using the BIC criterion to choose the subset of the variables as ordered by lasso. The software will treat factor levels as if they were distinct variables. I am thinking I should just keep the factor level pooling implied by the variable selection. I have read posts that are similar (eg. If a factor variable is to be dropped in model selection, should all levels be dropped simultaneously? If so, why?, Can I ignore coefficients for non-significant levels of factors in a linear model?) however there I'm not sure prediction was the main theme as opposed to inference (see 3 in the numbered list below).
There was a comment by Frank Harrell that factor level pooling compromises out of sample prediction. I wonder if this is necessarily true. Isn't the whole idea of doing variable selection in discriminative (as opposed to generative) modeling that there are variables whose estimated coefficients are non zero only due to sampling and thus out of sample they add noise (i.e. parsimony leads to better generalization)? Thus doesn't it make sense that if by penalized regression with an information criterion some of the levels of a factor get pooled, then that suggests those pooled levels dont have sufficiently distinct effects to determine that the estimated differences are not just due to sampling (i.e. artifacts). If they are due to artifacts then you are adding out of sample noise. In response to the objection that we shouldn't due "automated modeling" see 3 and 4 below. Some further points
- I am aware that choice of software procedure should be determined by statistical reasoning and not the other way around. But I do think there is reason in the above.
- I am aware that there is an effect of which level is chosen as the reference level. But doesn't that just mean you are only ever considering pools containing the ref level as opposed to considering all possible pools? To me considering all possible pools would just be even better instead of my suggestion being bad. I.e. I don't think it adds out of sample bias compared to not pooling at all.
- I work in credit risk analytics doing default rate modeling. The variables we get are often from credit reports where 1) I don't get a say in how the levels were chosen 2) I have no idea that the breakdown chosen by the credit reporting agency is the a priori best reasoned one for my specific application, 3) some factors have many levels where overfitting the effect of the factor is a real concern 4) the agencies essentially make a one size fits all choice of levels since their data is used by many many different people with different concerns 5) Consider that any level could be split into two more specific levels and that in my context I don't know why they chose the specificity they did and thus it may be that two or more levels are ones that i would have pooled and that they split to add specificity that i don't need.
- To maintain sensibility is it sufficient to a priori identify levels that I have reason to believe shouldn't be pooled and then look at the levels that are pooled and check if there are any that I object to (if my variable selection method and data are any good this should only rarely happen) and only if that occurs then use all the levels?
- Using Lasso with an information criterion isn't perfect but i believe a lot of the opposition in the related posts was centered on the term "significant" and what constitutes signifance. But I think a lot of the points boil down to issues with using significance level based variable selection because of things related to it presuming the test stats truly have a null f distribution
- Frank Harrell's comment seemed to center around prediction intervals not being the actual stated confidence level. In my application we don't typically have an opportunity to actually employ prediction intervals and if we did couldn't we just use bootstrapping to get a confidence interval for P(Y=1) (i.e we use logistic regression)?