I would like to perform logistic regression with few categorical variables. I started from fitting univariable models, just to check the relationship of independent variables in the absence of other variables. One of my variables (let's denote it as X1) has 18 levels, so I end up with 18 coefficients (1 intercept and 17 adjustments to reference level). Most of coefficients (12) is statistically significant based on Wald test, however the remaining 6 isn't. Important thing is that X1 could be easily split to 2 variables based on definition of the levels. Now I wonder if I should think about the choice of levels. Basically my question is - does different choice of levels could add predictive power? Does grouping/splitting these levels which coefficients are not statistically significant is valid operation?
1 Answer
The answer to this is "it depends". You should not go about grouping categorical variables just for the sake of improving model performance. The re-grouping of the categories should be based in some domain knowledge, and should be justifiable regardless of model performance. There is a chance that depending on the sparsity of some of the categorical levels that the effect did not reach significance, in which case it might make sense to group it with other like levels. However, there could also be a case that the level itself does contain some predictive power that you have just not uncovered in your sampling, at which point it might not make sense to group it with another factor level. Allow me to use an example to illustrate what I mean.
Let's pretend we are predicting the probability an animal would make a good pet. Our dependent variable is simply "Good Pet" and "Not Good Pet" which we code as 1 and 0 respectively. Now what if have just one independent variable that is a what the animal is, and it is specific to start. Let's make the levels "cougar", "lion", "tabby", "persian", "siamese", and "scottish fold" (all types of felines). Now pretend in your data that the "cougar" category is sparse, and just three responses (0, 0, 1). Chances are that the model will identify this level as not statistically significant. In addition, I also received few responses for "persian" cats, which also resulted in it not being statistically significant. Does it make sense to group these two categories even though adding the responses from "cougar" to the "persian" might result in a change in significance for that new level? Domain knowledge of cats would say "no", and that maybe if you want to do some grouping to see if it improves model performance it might make more sense to group the cats into "domestic" and "wild", or even "large" and "small". This eventual decision comes down to the importance of being able to evaluate the influence of the different factor levels. Is it important to your work to understand the potential increase in odds on your dependent variable between different levels of the categorical variable? If not, then you can begin grouping them to see if you have better classifications as a result.
Finally, I will note that you should be sure that you are interpreting the results of the model creating dummy variables for you. You have correctly identified in your question that one of the levels is absorbed into the intercept, but what is important to note is that insignificance of other levels are compared to that level absorbed into the intercept. To use my cat example again, if "lion" was represented by my intercept term I would almost expect that "cougar" would offer no significant effect over the one that is represented in the intercept by "lion".
Often questions like these come down to applying domain knowledge and reflecting on the type of information you are trying to gain from your study.
