1
$\begingroup$

I don't have much experience using logistic regression and therefore I wanted to pick your thoughts for what I should do with my model.

I am using logistic regression to see how do different variables affect my dependent variable. I have 30 independent variables and some of them are dummy variables. I have used the glmnet function to select the variables with which I can build my model and here comes my problem. As dummy variables I have for example the days of the week, however in the output from the glmnet function I can see that some of them are in the final model and some of them are not. Would you recommend using all the dummy variables in the model or should I use only the ones that improve the performance?

Or if you could recommend articles about application of logistic regression when you want to explain the relationships between the dependent and the predictors.

Thanks!

$\endgroup$
1
  • $\begingroup$ It seems reasonable to me to exclude them if you have confidence in the model. Let's say you had a categorical variable for colour with red, blue, green and yellow classes. If you made these into dummy variables, and glmnet excluded (i assume you mean had weights of 0 or close to it) blue, green and yellow you can just interpret the model as considering red or not red an important feature but being agnostic to different colours that aren't red. One thing I would be careful about though is if you dropped the first class when making your dummy variables. $\endgroup$
    – Dan
    Jan 31, 2018 at 14:13

1 Answer 1

0
$\begingroup$

It depends on the purpose of your model and whether there are potential interactions or effect modifications that are important in the full model.

There is one school of though that suggests only 'significant' variables should be retained in the model. This is logical from a degrees of freedom 'spending' argument, and even for maximising AIC.

There is another school of thought (that I am more aligned with) that views it as an experimental design issue. If you are including a categorical variable because it has a plausible link then, whether or not some levels predict the outcome, you should retain it in the model. In a multiple regression, all factors adjust for each other, so removing some levels of the effect will alter the estimation of other parameters. This is part of the problem with 'stepwise' model selection. Although this is a little different with lasso models, as the parameter is retained but the coefficient is zeroed.

Further point is, you have already run a model with all the parameters. Even if you drop some levels now, your 'significance' refers to the full model fit. You will be under-estimating your degrees of freedom and your measures of model fit won't be appropriate anymore.

I'd suggest that if some levels are significant, consider leaving the whole variable in.

Frank Harrell has written about this, with a focus on logistic models, and there is a lot of info in his Regression Modelling Strategies book, but his 1996 paper on this link is really useful. Section 3 is good to read before progressing with this problem.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.