Say you have an IV that can be k different categorical values, and you are trying to do logistic regression. To avoid multicollinearity, you dummy encode the IV into k-1 variables, with one of the categories being the reference group. represented by zeros in all k-1 columns. For all other categories, the corresponding beta value will be representative of the feature importance, but how do you determine the feature importance for the reference group? Also, how would you do this if Lasso regularization is used, and several betas were dropped?
1 Answer
$\begingroup$
$\endgroup$
The statement:
For all other categories, the corresponding beta value will be representative of the feature importance
isn't quite right. With dummy coding, the intercept term represents the outcome (log-odds in logistic regression; principle holds for all regression models) when the categorical predictor is at its reference level. The category-specific coefficients represent the differences of associations of the other category levels with outcome from that of the reference category. You can use the intercept and the category-specific coefficients to estimate the log-odds associated with each category.
So your sense of the problems this poses for predictor-selection schemes like LASSO is well founded, even worse than you might suspect at first. The relative penalization among more than 2 categories can depend on the choice of reference category and on whether the dummies are centered and scaled like continuous predictors usually are. See this page for example. (That's even a problem for penalized methods that don't do predictor selection, like ridge regression.) And some would argue that the entire categorical predictor (all levels) should be included or excluded together in variable selection, as done with group LASSO.
