Choose some of the dummy variables for building a logistic regression

Question

I don't have much experience using logistic regression and therefore I wanted to pick your thoughts for what I should do with my model.

I am using logistic regression to see how do different variables affect my dependent variable. I have 30 independent variables and some of them are dummy variables. I have used the glmnet function to select the variables with which I can build my model and here comes my problem. As dummy variables I have for example the days of the week, however in the output from the glmnet function I can see that some of them are in the final model and some of them are not. Would you recommend using all the dummy variables in the model or should I use only the ones that improve the performance?

Or if you could recommend articles about application of logistic regression when you want to explain the relationships between the dependent and the predictors.

Thanks!

Answer 1

It depends on the purpose of your model and whether there are potential interactions or effect modifications that are important in the full model.

There is one school of though that suggests only 'significant' variables should be retained in the model. This is logical from a degrees of freedom 'spending' argument, and even for maximising AIC.

There is another school of thought (that I am more aligned with) that views it as an experimental design issue. If you are including a categorical variable because it has a plausible link then, whether or not some levels predict the outcome, you should retain it in the model. In a multiple regression, all factors adjust for each other, so removing some levels of the effect will alter the estimation of other parameters. This is part of the problem with 'stepwise' model selection. Although this is a little different with lasso models, as the parameter is retained but the coefficient is zeroed.

Further point is, you have already run a model with all the parameters. Even if you drop some levels now, your 'significance' refers to the full model fit. You will be under-estimating your degrees of freedom and your measures of model fit won't be appropriate anymore.

I'd suggest that if some levels are significant, consider leaving the whole variable in.

Frank Harrell has written about this, with a focus on logistic models, and there is a lot of info in his Regression Modelling Strategies book, but his 1996 paper on this link is really useful. Section 3 is good to read before progressing with this problem.